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ABSTRACT 



This report presents a plan for an Educational 



Assessment and Accountability System for Hawaii. The purpose of this 
plan is to provide an analytical as well as a descriptive account of 
the condition of public tlucation in Hawaii. It is meant to be a 
vehicle to provide the means by which educational outcomes can be 
examined in light of contextual factors, resource utilization, and 
educational practices and policies. Chapter 1, " Introduction , " covers 
the following topics: (1) background information; (2) assessment 
needs; (3) national efforts in educational assessment; (4) education 
assessment and accountability system specifically for Hawaii; (5) 
related assessment and accountability activities; and (6) the purpose 
of this plan. Chapter 2, "Design," covers the following topics: (1) 
performance assessment; (2) context-input-output framework; (3) data 
sources; (4) focus on schools; (5) key questions and their policy 
relevance; (6) analysis of data; (7) accountability uses; and (8) 
flexible implementation. Chapter 3, "Implementation Plan," contains 
the following sections: (1) "Goals and Objectives"; (2) "Guiding 
Principles"; (3) "Critical Elements"; (4) "Organization and 
Administration"; (5) "Implementation Timeline"; (6) "Costs"; and (7) 
"Evaluation." Three tables, five figures, and a list of 18 references 
are included. Examples of current educational assessment related 
activities and a school status and improvement report are appended. 
(JS) 
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Executive Summary 



Introduction 



The present Implementation Plan for Educational Assessment and 
Accountability has been prepared for submission to the 1990 
Sixteenth Legislature. The plan has been prepared as a response 
to S.B. No. 1856 of the Fifteenth State Legislature, 1989, and to 
the Legislative Auditor's final report on the Evaluation of the 
Administrative Flexibility Legislation Affecting the Department of 
Education and University of Hawaii (1989). The Implementation 
Plan timeframe spans the years 1990-1994. 

Background 

The importance of a healthy, high quality educational system 
grows more critical by the day. The economic and social 
consequences of educational stagnation can be enormous for an 
individual as well as a nation. In Hawaii, as in many states, there 
is a growing concern over the quality of the schooling process and 
public education in particular. 

The past decade has witnessed a host of signals that may be 
serving as harbingers of serious economic and geo-political 
setbacks for our nation. The publication of A Nation At Risk 
early in this decade (National Commission on Excellence in 
Education, 1983) and its capstone remark referring to the ". . . 
rising tide of mediocrity [in American education]" preceded a 
voluminous list of commission and task force reports and research 
studies that share a common theme: the condition of American 
education grows more deplorable with each passing year. 
International comparisons of student performance on math and 
science tests reveal consistent results that place the U.S. a distant 
"also ran" behind Japan, Germany, England, and most other 
developed, democratic countries. 

Other signals of educational decay include a long-term decline in 
the Scholastic Aptitude Test (a measure designed to predict 
college performance), employer dissatisfaction with applicant basic 
skills, college faculty feedback on freshman computing, reading, 
writing and analytical skill deficiencies, and a shortage of math, 
science and technology graduates. 
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In Hawaii, similar concerns have been expressed. Additional 
concerns somewhat unique to our island community include the 
overall poor student performance in the area of verbal skills on 
nationally normed standardized tests, and the loss of some of the 
islands' most brightest and promising youth via the "brain drain." 

Clearly, there is a need for an attentive and involved citizenry to 
foster educational reform. There does appear to be widespread 
consensus to take action to improve the quality of public 
education and the performance of our public schools. Less clear 
is the choice of specific actions to take, and how we wil! assess 
the effects of such actions. 

Increasingly, educators and policymakers are turning to 
educational assessment and accountability systems as a 
decisionmaking tool and as a vehicle for public accountability. 
The renewed national and state-level interest in education 
assessment/accountability systems reflect the shared concern that 
new policies and follow-up actions be based on reliable 
information. For policymakers and the general public, better 
information is seen as a key to improving schools' performance 
and holding them accountable. For educators, better information 
is viewed as a key to more effective planning and more efficient 
operations. 

Need Statement 

A continual review of educational quality is necessary to make 
systematic improvements. The Educational Assessment and 
Accountability System for Hawaii public education is not a 
panacea, but can be highly useful to policymakers with 
responsibility for public education. Unfortunately, policymakers 
often lack the kinds of information critical to the decisionmaking 
process. Available data typically does not allow policymakers to 
identify the sources of problems with enough precision to 
formulate effective policy (Shavelson et al., p. v, 1987). 

Educators, too, share a similar paucity of useful information 
necessary for decisionmaking on an operational level. Here, too, 
an educational assessment and accountability system, properly 
designed, and given sufficient resources, can play a pivotal role in 
monitoring and facilitating improvement efforts in education. 
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System Overview 



The basic components of an Educational Assessment and 
Accountability System consist of a set of educational indicators 
linked to an analytical model. Multiple indicators are required to 
provide a representative assessment of the schooling process and 
educational outcomes. 
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A wide range of formal assessment and accountability mechanisms 
already exist within the Department. Some examples are 
monitoring for compliance with civil rights and equal opportunity 
laws; management audits; financial audits; special analytic studies 
and other Program Planning Budgeting (PPB) system 
requirements; personnel evaluations; program monitoring and 
program audits; program evaluations; curriculum reviews; school 
accreditation; and student testing. 

Clearly, each of these mechanisms serves an important role in 
holding educators and the Department accountable. Each 
mechanism, however, is aimed at a specific purpose and often 
uses different information. 



The current assessment and accountability plan is based on the 
straightforward idea of combining and building on selected data 
elements from existing assessment and accountability mechanisms 
in order to broadly but comprehensively examine schools' 
performance outcomes. Important to note is that the intent of 
the current plan is not to integrate the various assessment and 
accountability activities already in place, but rather to integrate 
selected information . 



Purpose Statement 

The Educational Assessment and Accountability System proposed 
herein is intended to provide an analytical as well as a descriptive 
account of the condition of public education in Hawaii. It is, by 
design, a vehicle to provide the means by which educational 
outcomes can be examined in light of contextual factors, resource 
utilization, and educational practices and policies. 
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The major purposes of the Educational Assessment and 
Accountability System for Hawaii's public schools are: 



• to provide information about schools' performance for public 
accountability; 



• to inform educational policy development; and, 

• to improve educational quality by influencing local practice and 
improvement efforts. 

The primary information users or audiences for the information 
are the general public, educational policymakers, and educators. 

We view the Educational Assessment and Accountability System 
as adaptive to users' information needs and evolving over time. 
Early experience with the system, including comments and 
criticism of the present plan, will help guide its future 
development. 



Design 

Creating an educational assessment and accountability .^vstem 
requires difficult choices on issues involving complex trade-offs. 
There does not exist an established model system that can be 
simply adopted (Oakes, 1986, p. 8). The design problems 
encountered are as much ecological (that is, organizational, 
structural, and political) as technical. The State Accountability 
Study Group's report, Creating Responsible an d Responsive 
Accountability Systems (U.S. Department of Education, Office of 
Educational Research and Improvement; September 1988) 
emphasizes that "Each State must design an accountability strategy 
that supports its own educational goals and needs and is 
consistent with its own political traditions" (p. vii). 

Performance Assessment: Focus on School Outcomes 

Educational assessment systems are the foundation for educational 
accountability systems. Richards (1988) has studied educational 
assessment systems in many states and school districts throughout 
the nation. He describes "three prototypical models that differ 
according to their purpose: monitoring for regulatory compliance, 
monitoring for instructional diagnosis/remediation, and monitoring 
for school performance outputs" (p. 107). Of these three types, 
monitoring for school performance outputs appears most 
appropriate as the foundation for Hawaii's educational 
accountability system. In the education accountability literature, 
this type of assessment or monitoring system is commonly referred 
to as performance assessment. The functional objective of the 
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performance assessment approach is to promote school 
performance (e.g., student achievement) by focusing oversight on 
school outcomes. 

Indicators: Context-Input-Output Framework 

The basic building blocks of a performance-based educational 
assessment and accountability system are indicators. Indicators 
are statistics that serve as gauges to inform policymakers, 
educators, and the public about the condition of the educational 
system and how it is changing. The OERI State Accountability 
Study Group's report (1988) notes, however, that "Accountability is 
a blunt tool unless policymakers, educators, and the public have 
information that allows them to determine the likely sources of a 
problem and find clues about how to fix it" (p. 7). Thus, what is 
needed is not just information about distinct elements of the 
educational system but also information about how the relevant 
elements work together to affect educational outcomes. In 
essence, model building is needed to identify the central features 
of the educational system. The basic analytical model for 
educational assessment and accountability we have adopted 
encompasses three general components: CONTEXT. INPUT, and 
OUTPUT. 

OUTPUT indicators such as graduation rates, grade point average, 
or performance on achievement tests gauge the health of public 
education. These outcome measures are the crux of the 
assessment and accountability system. Outcome measures, 
however, are necessarily viewed in CONTEXT with the community 
setting, family demographics, and the students' readiness for 
school. In addition to context variables, INPUT indicators 
reflective of policy guidelines and the schooling process itself are 
examined. Some examples of input indicators include course 
offerings, course requirements, staff allocation, teaching methods, 
arid finances. 

It is critical that the educational assessment and accountability 
system include multiple output or outcome measures that range 
from achievement test scores and attendance rates to disciplinary 
incidents and graduates' performance in the first year of college. 
Multiple output indicators are needed in order to represent 
adequately the overall results of the educational system. 

For policymakers, an understanding of relationships between 
inputs and a school's outcomes, as conditioned by the 
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school-community context, will be most useful. The intent of 
using the context-input-output indicators framework is for its 
potential explanatory and diagnostic value, that is, as a guide to 
how the various elements of the educational system might work 
together to affect outcomes. 

Data Sources 

Several existing data sources, together with new sources currently 
under development, should provide a basis of sufficient scope and 
depth for an initial set of indicators. Of these, the School 
Profiles is the most comprehensive and the single best data source 
for indicators development. Generated by the Department's 
Information System Services Branch, the School Profiles contain 
multi-year statistical information, updated annually, for each 
school. Drawing upon data extracted from other major data 
sources, School Profile data are available in the areas of student 
achievement, student demographics, student behavior, teacher 
demographics, and school-community characteristics. 

Other existing data sources or assessment-related activities that 
will be immediately useful for outcome indicators include the 
Hawaii State Test of Essential Competencies, the College Board's 
Scholastic Aptitude Test, and the Graduate Follow-Up Survey. 

Focus on Sc hools and Reporting 

There is ample research and experience to indicate that individual 
schools are the basic unit of improvement in public education. 
Thus, the best place to observe the interaction of state, district 
and school policies, as they affect students and practitioners, is at 
the school level. Clearly, the choice of a focal level has both 
technical and policy implications. The unit of analysis should be 
comparable to the level at which policy impact is expected; most 
often, this will be the school level. 

Educational assessment and accountability reports, as presently 
conceptualized, would be produced for each school, A state 
summary of information selected from the school reports, together 
with additional information relevant to a state-level report (e,g., 
National Assessment of Educational Progress results, selected 
findings from a new Survey of Public Education in Hawaii), would 
also be produced. Reporting will emphasize the school as the 
primary unit of accountability, while providing policymakers with 
overall state results. 
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Key Question s and Polic y Relevance 



Three key questions underlie many recurrent policymaking issues 
in Hawaii's public school system. 

• What are the overall condition, performance, and progress of 
the public schools? 

• What are the accomplishments and shortcomings of the 
educational system? Where are adjustments needed? 

m 

• To what extent are intended effects, including those of major 
education initiatives/reforms, being achieved? 

Fundamental to the present design is the assumption that the 
resulting information will be used with the intent of improving the 
quality of public education in Hawaii. We believe the three key 
questions cited above serve as a central guide to providing 
policy-relevant information. 

Analysis 

The analysis component of the present design must include three 
analytic functions necessary to operationalize the use of indicators 
for the purpose of providing policy-relevant information about 
schools' performance: (1) describe performance to answer the 
question "What is happening?"; (2) relate performance to inputs 
and context variables to answer the question "Why might it be 
happening?"; and (3) compare performance to answer the question 
"Is it adequate?" 

Descriptive, relational, and comparative analyses tend to form a 
sequence of increasing analytic complexity, with later types 
building on the results from earlier ones. We envision the initial 
years' assessment/accountability reports as based largely on the 
results of descriptive and relational analyses, but also containing 
findings on a few selected comparative analyses of outcomes. The 
comparative analyses, initially, would likely use schools' past 
performance and/or national norms as bases for comparison. 

Accountability Uses 

The design of the present system utilizes a performance 
assessment approach; it serves as a tool for improving public 
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education by attempting to better inform policymakers, educators, 
and the general public. Simply said, the design is 
outcome-oriented and it seeks improvement by focusing attention 
and discussion on educational outcomes. 

Unless assessment results are used in some instrumental way for 
school improvement, accountability has not been realized. The 
following are some uses we propose should be made of the 
assessment reports. 

Use assessment findings/reports to: 

• Provide for accountability to the general public; 

i 

• Provide for accountability to policymakers and to assist the 
development of educational policy; 

• Guide the Department's program planning snd budget 
development; and, 

• Guide School Improvement Plan development. 
Flexible Implementation 

There is ample evidence to indicate that ineffective 
implementation is a cause of the failure of many social policies 
and programs to achieve their intended effects. Be r man (1980) 
noted that a mismatch between implementation approach and 
context of the situation in which the initiative is to be 
implemented can actually worsen the very implementation 
problems the approach is supposed to circumvent. For those 
implementation areas of the Educational Assessment and 
Accountability System that are highly dependent upon professional 
judgement rather than routine application of simple 
decisionmaking rules, adaptive rather than pre-programmed 
strategies will work best. 
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Implementation Plan Summary 

This section describes the plans developed by and implementation 
steps taken by the Department, and joint activities conducted by 
the Department and the University of Hawaii that relate directly 
to educational assessment. 
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Goals 

The following four goals define the functional scope of the 't^eS^/Sicdri^ 
Educational Assessment and Accountability System. 

AocoMtabOitjr System. 

• Establish a statewide system of educational assessment and 
accountability to systematically examine the health and 
quality of Hawaii public education. 



• Institute publir accountability through periodic reports on 
public education to thr community-at-large (parents, 
businesses, taxpayers). 

• Inform educational policymakers and educators about the 
condition, performance and progress of Hawaii public 
education. 

• Work collaboratively with the University of Hawaii system to 
coordinate educational assessment activities between the 
Department and higher education. 



Specific implementation objectives corresponding to these goals 
are given in the full report. 



Guiding Principles 

The following guiding principles serve to further direct the 
development and implementation of assessment/accountability 
activities: 



• Information produced by the Educational Assessment and 
Accountability System should be relevant to the development 
and revision of educational policy and to the assessment 
questions directly related to policy issues. 

• The results and report formats should be clear and easily The R»it> wi report 
understood by policymakers, educators and the community-at- S^^tilSUbJ™' 
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• An independent advisory panel should be established to 
assist in the development and review of the Educational 
Assessment and Accountability System. 
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• The Educational Assessment and Accountability System 
should be based on an analytical model that serves as a 
conceptual framework for identifying, developing and linking 
a set of educational indicators. 



• Assessments of educational quality should dju be based 

solely on single indices of performance such as standardized 
tests; rather, multiple indicators with analytical links among 
educational input, context, and output elements should be 
used. 



• The Educational Assessment and Accountability System 
should be adaptable, evolving over time to meet future 
assessment needs necessitated by changes in the public 
education delivery system itself (e.g., School/Community- 
Based Management). 

• The school should serve as the fundamental unit of analysis, 
consistent with its role as the primary agent of change. 

• The Educational Assessment and Accountability System 
should utilize and integrate extant data to the fullest extent 
possible to reduce paperwork and data collection burden on 
school personnel. 

Implementation of Critical Elements 

There are seven critical elements involved in the development and 
implementation of the Educational Assessment and Accountability 
System. Each element is briefly discussed below. The full report 
contains an implementation schedule that details the 
implementation timeline for each element. 

(1) Policy Issues and Assessment Questions 

The Educational Assessment and Accountability System is 
intended to be more than a mere collection of data. 
Foremost among our guiding principles is the requirement 
that information be relevant to policymaking and public 
accountability. 

Specific assessment questions subsumed under each policy 
issue area are essential to specifying the type and scope of 
data required. The following is an example of a possible 
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policy issue and the related assessment questions that might 
be addressed, 



Policy Issue: Is the scope ami sequence of curriculum 

offerings adequate in preparing graduates for 
subsequent study in higher education? 

Assessment Questions: 

1. What percent of applicants wfw graduate from Hawaii 
public schools are accepted at Hawaii's 4-year colleges? 

2. Wliat percent of F.awaii public school graduates receive 
advance placement credits, or course waivers as freshmen 
at UH Manoa? 

3. Ho\v many Hawaii public school graduates entering the 
UH system are required to enroll in remedial courses in 
language arts? in math? 

4. Which indicators best predict freshmen performance in the 
UH system? 



(2) Indicator System 

Crucial to the establishment of the Educational Assessment 
and Accountability System is the development of an indicator 
system. A lengthy yet far from exhaustive list of such 
indicators has been drafted in a form of a taxonomy 
(Educational Assessment and Accountability Project Working 
Paper #2, A Taxonomy of Indicators for Educational 

Assessment and Accountability; August 1989 (Draftj). 

The taxonomy lists potential indicators by assessment 
domains. For example, context indicators are organized 
under two assessment domains: demographics (1.10) and 
early education experience/readiness (120). We have 
underscored potential to emphasize that this taxonomy was 
designed to embrace much more than would be feasible in 
an actual educational assessment and accountability system. 
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The taxonomy is specifically intended to serve as r* resource 
document to guide further efforts toward implementing a 
useful, representative, and parsimonious set of educational 
indicators. A set of initial indicators is proposed in the full 
report; future developmental steps are outlined therein as 
well. 

Several new data sources are currently under development or 
being planned. Some additionally useful educational 
indicators, based on these new data sources, are expected to 
become available within the next one to two years. Most 
important among the new data sources are: 

• University of Hawaii's Freshman Report of High School 
Graduate: 

• National Assessment of Educationu' Progress (NAEP) 

• School Climate/Effective Schooling Survey 

• Survey of Public Education in Hawaii 

• Instructional Practices Surveys (Testing Supplement) 

As work progresses on the development of the initial set of 
indicators and new data sources, a number of measures will 
be scrutinized specifically for potential use as "leading 
indicators" (Oakes, 1986). The concept of a leading 
indicator (borrowed from econometrics) is an intriguing one 
for educational assessment and accountability. The intent of 
establishing a small array of leading educational indicators is 
to provide a very concise account of the condition and 
direction of public education, and to provide, if possible, an 
"early warning system" for educational policymakers. 

(3) Data Collection Plan 

Data Definition . The initial set of indicators will require 
operational definitions for each data element of interest. 
Even seemingly "basic" indicators such as attendance rates, 
graduation rates and dropout rates are often inconsistently 
defined and reported across states, districts, and schools. 

Procedures . Detailed data collection procedures will be 
drafted during the first few years of the implementation 
plan. As developmental work progresses in other related 
areas (e.g., prioritizing policy issues and assessment 
questions, and ope rationalizing data element definitions), 
data collection instruments and procedures will be refined. 
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Implementation . The first step in data collection will be the 
development and definition of an initial set of context, input 
and output measures. The second step will involve the 
development and refinement of data collection forms and 
instruments to obtain information not available from existing 
sources. Data collected will be collated, tabulated and 
stored in computer database files created for the Educational 
Assessment and Accountability System. 

These activities require a third step in the data collection 
scheme: the establishment of a database storage 'and 
retrieval system. The fourth step, and an especially 
important aspect within the data collection design, is a 
quality assurance component to provide validity checks and 
data quality controls. 

Measures contained in the School Status and Improvement 
Report can serve as an interim set of educational indicators. 
Future data collection from the school level is not 
anticipated to be much different from that utilized for the 
current School Status and Improvement Report. 

(4) Standards 

The area of standards setting is indeed difficult. But the 
need to establish educational standards such as minimally 
acceptable performance expectations (i.e., minimal standards) 
based on strong rationale rether than impulse seems more 
critical than ever. School, district and state officials may not 
be so adamantly opposed or fearful of standards if results 
are truly useful, and are analyzed cautiously and reported 
fairly. 

The first year of the implementation plan will include 
preparations for convening a special study group on 
standards setting. Due to the difficult and complex nature 
of the task, the Department will likely conduct a nationwide 
search to acquire the services of professionals with expertise 
in this area. A number of highly reputable institutions, 
including the RAND Corporation and several major 
universities and educational agencies has expressed a 
willingness to help Hawaii with standards setting. 
Simultaneously, the Educational Assessment and 
Accountability Project staff will initiate work on reviewing 
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pertinent literature on establishing standards during this first 
year. The second and ensuing years will be devoted to 
preparation of materials necessary for use by the special 
study group, documentation of study group recommendations, 
and field-testing of educational standards for selected 
indicators prior to formal adoption. 



(5) Analysis 



How can the adequacy o* a school's performan ce on a set of 
outcome indicators be operationally defined? A school's 
performance is obviously affected by the availability of basic 
resources such as personnel textbooks, and classrooms. 
However, it is not only the amounts but also the ways in 
which the inputs are utilized that is important. 

A school's performance can also be affected, positively or 
negatively, by context factors (e.g., socioeconomic status of 
the school's community). Most context factors are ordinarily 
beyond the immediate control of either the school or the 
school system. Nonetheless, context factors differ 
considerablv from school to school and do affect educational 
outcomes. Any fair assessment of a school's performance 
must somehow take into account or control for variations in 
context between schools. 

One proposed analytic approach includes the technique of 
multiple linear regression (MLR). MLR is a general 
statistical technique through which the relationship between a 
criterion (dependent) variable and a set of predictor 
(independent) variables can be analyzed. For present 
purposes, the most important uses of the technique are: (1) 
to identify the best sets of context and input (independent) 
variables that contribute to the prediction of each given 
outcome (dependent) variable; and (2) to control for the 
confounding of context variables in order to evaluate the 
specific contributions of inputs to outcomes. 

Several states currently utilize MLR approaches as part of 
their assessment/accountability systems. A few states (e.g., 
California, South Carolina) rely extensively on variants of 
two prototypical MLR approaches described in the full 
report. 
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Considering the experience of other states that have used 
MLR in their assessment/accountability systems, it seems 
advisable not to base Hawaii's educational assessment 
methodology on MLR analysis alone. Rather, the proposed 
methodology includes MLR analysis but as only one part of 
a comprehensive sequence of analysis. Following is a brief 
sketch of the proposed assessment methodology. 

PART 1 - Current Performance Compared to State 

Standards 

For each outcome indicator, compare a school's current 
performance with the established state standard or 
criteria. Simple, easily understood information is 
provided by such comparisons. The assessment 
information directly answers the basic question "Is the 
school's performance adequate?" 

PART 2 - Current Performance Co mpared to Past 
Performance 

For each outcome indicator, compare a school's current 
performance with its past performance. This 
assessment information directly addresses the question 
"Is the school's performance improving?" 

PART 3 Current Performance Compared to That of 
Similar Schools 

For each outcome indicator, use MLR analysis to 
compare a school's actual performance with its' 
"expected" performance. This assessment information 
addresses the question "Is the school's performance as 
good as can be expected, given its context and inputs?" 

There is much exploratory and developmental work to be 
done in this area. The Context-Input-Output analytic model 
and the establishment of an indicators system provide the 
basic ingredients necessary to initiate the Educational 
Assessment and Accountability System. The first year of 
implementation activities involving the development of an 
initial set of educational indicators and work on standards 
setting are crucial prerequisites to the proposed analytical 
methodology. Also important are the exploratory studies 
needed to ascertain the utility of a special subset of "leading 
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indicators," and the actual performance of particular analytic 
techniques. 



(6) Reporting 

Reports will be produced to summarize information at both 
the school and state levels. The initial statewide report is 
scheduled to be released in early 1990. The bulk of that 
report will be based on information from the newly 
implemented School Status and Improvement Report (SSIR) 
[a revised version of the School Performance Report Card]. 
The SSIR is expected to serve as an interim school report 
prior to modification into a more comprehensive Educational 
Assessment and Accountability report format. 

The purpose of the state-level report is to appraise the 
condition, performance and progress of Hawaii public 
education. Efforts will be made to identify overall trends 
and to assess the general impact of educational initiatives 
and reforms. 

The contents of future state-level Educational Assessment 
and Accountability Report will include summaries of school 
performance by district and state. These summaries will 
report results from analysis that reflects the influence of 
contextual factors and resource utilization in addition to 
outcomes. The kinds of information presented will range 
from summaries of community demographics and school 
readiness assessments to achievement levels and outcomes 
beyond high school graduation. 

Development of new, prototype school and state-level 
Educational Assessment and Accountability reports will begin 
in the second year of implementation. The report formats 
are expected to be slightly different for elementary and 
secondary schools. 

It is also planned that a general report to the public be 
issued biennially beginning 1995. Contents of that report 
will include highlights of educational assessment and 
accountability findings and a special feature section covering 
topics of current interest. 
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(7) Informed decisionmaking 



The basic tenet that underlies the educational assessment 
and accountability concept is that overall improvements in 
education can be greatly facilitated by providing key 
information in the right place at the right time. The 
ultimate utility of information made possible through the 
Educational Assessment and Accountability System is its 
potential for informed decisionmaking at school, district and 
state levels. Parents and the business community, too, 
should find the reports and other end-products useful. 

The state-level, school-level and public accountability reports 
comprise the major deliverables produced by the Educational 
Assessment and Accountability Project. There are, however, 
a number of other possibilities that may benefit 
decisionmakers and policymakers. Two such possibilities are 
mentioned here. 
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First, the Educational Assessment and Accountability 
database and its affiliated data sources will have the 
potential to provide supplemental information. Special 
requests for technical assistance or database retrieval of 
particular pieces of information may be a possibility once the 
basic Educational Assessment and Accountability System is 
in place and data quality is improved. 

Second, a number of current or special issue areas will 
inevitably require further in-depth study than is possible with 
educational assessment and accountability information. It 
may be possible, indeed desirable, to conduct special studies 
on current educational topics of interest. For example, there 
is considerable interest in obtaining a comprehensive and 
accurate assessment of public school graduates' employment 
patterns. The Department of Education, Department of 
Labor and Industrial Relations, University of Hawaii and the 
business community, in particular, would all benefit if 
employment data on recent graduates were made available 
in a timely manner. 



Organization and Administration 



No major reorganization will be required within the Department 
of Education to implement educational assessment and 
accountability activities. The Educational Assessment and 
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Accountability Project staff will be incorporated within the existing 
Evaluation Section, under the Planning and Evaluation Branch, 
Office of the Superintendent. 

The administration of the Educational Assessment and 
Accountability Project will be under the general direction of the 
Evaluation Section's Administrator. The current line of authority 
within the Office of the Superintendent will remain intact. 

Moreover, an Advisory Panel on Educational Assessment and 
Accountability will be established in the first year of 
implementation. This independent panel will consist of 
policymakers, educators, parents and the community-at-large. The 
primary role of the Panel will be to periodically review and 
provide feedback to guide the implementation and long-term 
direction of the Educational Assessment and Accountability 
Project. 

During the timeframe spanned by the Implementation Plan, 1990- 
1994, the average annual cost of implementing the Educational 
Assessment and Accountability System is estimated at $151,864.00. 
Budget details may be found in the complete report that follows. 
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Introduction 



The present Implementation Plan for Educational Assessment and Accountability 
has been prepared for submission to the 1990 Sixteenth Legislature. Tie 
present document has been prepared as a response to S.B. No. 1856 ot the 
Fifteenth State Legislature, 1989, and to the Legislative Auditor's final report on 
the Evaluation of the Administrative Flexibility Legisla tion Affecting the 
Department of Education and University of Hawaii (1989). The Implementation 
Plan timeframe spans the years 1990-1994, and includes first, information on the 
impetus, necessity for and purposes of an Educational Assessment and 
Accountability System for Hawaii public education. Second, the design of the 
Educational Assessment and Accountability System is explained in some detail to 
describe the conceptual framework, model, and data analytic approaches required 
to compile, analyze, and synthesize information on educational outcomes, 
contextual factors, and policies and practices. Finally, seven critical elements of 
the Educational Assessment and Accountability System are outlined and briefly 
discussed in conjunction with implementation goals, objectives, system 
organization and administration, timeline, and costs. 

A. Background 

The importance of a healthy, high quality educational system grows more 
critical by the day. The economic and social consequences of educational 
stagnation and neglect can be enormous for an individual as well as a 
nation. In Hawaii, as in many states, there is a growing concern over the 
quality of the schooling process and public education in particular. 

The past decade has witnessed a host of signals tha* may be serving as 
harbingers of serious economic and geo-political setbacks for our nation. 
The publication of A Nation At Risk early in this decade (National 
Commission on Excellence in Education, 1983) and its capstone remark 
referring to thv . . rising tide of mediocrity [in American education]" 
preceded a voluminous list of commission and task force reports and 
research studies that share a common theme: the condition of American 
education grows more deplorable with each passing year. International 
comparisons of student performance on math and science tests reveal 
consistent results that place the U.S. a distant "also ran" behind Japan, 
Germany, England, and most other developed, democratic countries. 

Other signals of educational decay include a long-term decline in the 
Scholastic Aptitude Test (a measure designed to predict college 
performance), employer dissatisfaction with applicant basic skills, college 
faculty feedback on freshman computing, reading, writing and analytical 
skill deficiencies, and a shortage of math, science and technology graduates. 

In Hawaii, similar concerns have been expressed. Additional concerns 
somewhat unique to our island community include the overall poor student 
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performance in the area of verbal skills on nationally normed standardized 
tests, and the loss of some of the islands' most brightest and promising 
youth via the "brain drain." 

There are early signs of a concerted effort to make Hawaii public 
education a priority at the policymaking level. The State Legislature, 
Governor's office, State Board of Education, and local business community 
have all taken an increasingly proactive role in reviewing and implementing 
educational policy and practice. Of special interest is the Governor's call, 
in his 1989 State of the Slate address, ". . . to enter the 21st century with a 
public education system that is second to none." There are some 
overarching similarities between the Governor's emphatic plea and the 
National Science Board's (Commission on Precollege Education in 
Mathematics, Science, and Technology) primary goals set in 1983: ". . . to 
provide all the nation's youth with a level of education in mathematics, 
science, and technology that is of the highest quality attained anywhere in 
the world ... by 1995." (Shavelson et al., 1987). 

Clearly, there is a need for an attentive and involved citizenry to foster 
educational reform. There does appear to be widespread consensus to take 
action to improve the quality of public education and the performance of 
our public schools. Less clear is the choice of specific actions to take, and 
how we will assess the effects of such actions. 

At present, there are a number of reform movements that directly impact 
Hawaii public education. Significant among these are the Hawaii Business 
Roundtable Study that culminated with the Berman-Weiler Report in 
Spring 1989, the School/Communi. /-Based Management process scheduled 
for implementation in up to 30 schools in School Year (SY) 1989-90, 
substantial teacher pay raises, generous capital improvement funds to 
upgrade school facilities, statewide expansion of parent-community 
networking centers, and attempts to more widely institute pre-school 
education, schools of choice, juvenile delinquency prevention programs, and 
after-school care programs. 

These and a host of other efforts to improve educational quality are varied 
and can be of differential value. How do we determine which efforts are 
worth pursuing? Which efforts should be discontinued? How do we know 
if our schools are making significant progress? And how can we avoid 
haphazard conglomerations of programs of dubious merit? 

These difficult questions are being posed in virtually every state. 
Increasingly, educators and policymakers are turning to educational 
assessment and accountability systems as a decisionmaking tool and as a 
vehicle for public accountability. The renewed national and state-level 
interest in education assessment/accountability systems reflect the shared 
concern that new policies and follow-up actions be based on reliable 
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information. For policymakers and the general public, better information is 
seen as a key to improving schools' performance and holding them 
accountable. For educators, better information is viewed as a key to more 
effective planning and more efficient operations. 

Need Statement 

A continual review of educational quality is necessary to make systematic 
improvements. The Educational Assessment and Accountability System f or 
Hawaii public education is not a panacea, but can be highly useful to 
policymakers with responsibility for public education. Unfortunately, 
policymakers often lack the kinds of information critical to the 
decisionmaking process. Available data typically does not allow 
policymakers to identify the sources of problems with enough precision to 
formulate effective policy (Shavelson et al., p. v, 1987). 

Educators, too, share a similar paucity of useful information necessary for 
decisionmaking on an operational level. Here, too, an educational 
assessment and accountability system, properly designed, and given sufficient 
resources, can play a pivotal role in monitoring and facilitating 
improvement efforts in education. 

National Ef forts in Educational Assessment 

At the encouragement of the Legislative Auditor (1989), the Department of 
Education conducted a review of educational assessment programs in other 
states (Educational Assessment and Accountability Project Working Paper 
#1, Review of Educational Accountability Systems in Other Stages: May 
1989 [Draft]). Our findings reaffirm the Auditor's suspicion that 
educational assessment "systems" are widely disparate and can take many 
forms. Some states simply reorganized or expanded statewide testing 
program activities and called the "new" program an educational assessment 
system. A number of states utilized a compliance approach to ascertain 
whether school districts are complying with state regulations. Several states 
have adopted a performance assessment type of educational assessment. At 
least one state utilized a diagnostic model of educational assessment where 
information is aggregated at the student rather than the school level. 
While none of these educational assessment systems are substantively 
similar, all share a common mission: To assess the condition of public 
education. 

Having examined existing state educational assessment systems elsewhere, 
and having reviewed the strengths, weaknesses and implementation 
requirements associated with each major type, we next attempted to design 
an educational assessment approach appropriate to assessing the condition 
of public education in Hawaii. 



An Educational Assessment and Accountability System For Ha waii 



The basic components of an Educational Assessment and Accountability 
System consist of a set of educational indicators linked to an analytical 
model of education. Multiple indicators are required to provide a 
representative assessm : the schooling process and educational 
outcomes. Admittedly, "accountability is still a largely untested concept in 
the field of education" (Anderson, Ball, Murphy, and Associates, 1975), 
even though various accountability mechanisms have been borrowed from 
the legal and financial fields for several decades. Educational assessment, 
too, in concept, if not name, has been in existence for some time. A case 
in point is Hawaii's Project ASSESS, a promising project that in 1973 
attempted to address many of the same concerns relevant today with 
relatively sophisticated analytical techniques (Hawaii State Department of 
Education, 1973). Especially noteworthy were attempts to provide school 
level analysis and practical recommendations for improvement in a report 
layout similar to the California school performance monitoring system 
which has received much attention recently. 

The renewed emphasis on educational accountability and assessment 
mechanisms seems directly related to concerns about educational quality 
and school reform effectiveness. Most noteworthy is the current focus on 
policy-relevant information. Fundamental to the present system is the 
ability to analyze how contextual variables, together with educational 
policies and practices, influence student outcomes. 

R elated Asses^nent an d Accountability Activities 

A wide range of formal assessment and accountability mechanisms already 
exist within the Department. Some examples are monitoring for 
compliance with civil rights and equal opportunity laws; management audits; 
financial audits; special analytic studies and other Program Planning 
Budgeting (PPB) system requirements; personnel evaluations; program 
monitoring and program audits; program evaluations; curriculum reviews; 
school accreditation; and student testing. Appendix A contains a list of 
educational assessment related activities currently supported or incorporated 
into public education programs. Clearly, each of these mechanisms serves 
an important role in holding educato- and the Department accountable. 
Each mechanism, however, is aimed a specific purpose and often uses 
different information. 

The current assessment and accountability plan is based on the 
straightforward id-a of combining and building on selected data elements 
from existing assessment and accountability mechanisms in order to broadly 
but comprehensively examine schools' performance outcomes. Also, the 
present plan is more closely related to some of the existing accountability 
mechanisms (e.g., student testing, program evaluation, curriculum reviews) 



than to others (e.g., compliance monitoring, fiscal or management audits). 
Important to note is that the intent of the current plan is not to integrate 
the various assessment and accountability activities already in place, but 
rather to integrate selected information . 

Links between the Educational Assessment and Accountability Syctem and 
extant accountability mechanisms will take time to develop and grow. It is 
anticipated that two important consequences are likely to emerge over time. 
First, findings from the new system may trigger special follow-up studies 
from the other accountability mechanisms. For example, program 
evaluations or curriculum reviews might be needed in order to obtain more 
detailed information about a particular area. Second, problems with the 
quality of some data elements are likely to improve over time as multiple 
users share the same data, become more knowledgeable abou data 
limitations and probiems, and begin to demand better quality nformation. 

In addition, it is expected that the school-level assessment and 
accountability reports will have many uses. Reviewed across several years, 
the school reports could be us«d to assess the annual and cumulative 
results of each school's improvement efforts. Similarly, some (but not all) 
of the benefits expected of schools adopting the School/Community-Based 
Management approach could be monitored via the school-level assessment 
and accountability reports. The school reports might also be made 
available to visiting teams conducting schooi accreditations. A set of the 
school's recent reports would be a useful and credible supplement to the 
mostly qualitative information contained in the school's own Self-Study 
report. Other worthwhile uses of the reports may be suggested or become 
evident over time. 

Purpose Statement 

The Educational Assessment and Accountability System proposed herein ii 
intended to provide an analytical as well as a descriptive account of the 
condition of public education in Hawaii. It is, by design, a vehicle to 
provide the means by which educational outcomes can be examined in light 
of contextual factors, resource utilization, and educational practices and 
policies. 

The major purposes of the Educational Assessment and Accountability 
System for Hawaii's public schools are: 

• to provide information about schools' performance for public 
accountability; 

• to inform educational policy development; and, 
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• to improve educational quality by influencing local practice and 
improvement efforts. 

The primary information users or audiences for the information are the 
general public, educational policymakers, and educators. Included within 
the policymaking group are the Governor and his staff, members of the 
Hawaii State Legislature, and members of the Board of Education. 

There are several different ways an educational assessment and 
accountability system can be designed. Also, many variations are possible 
on planning the steps to implementing any given design. The present 
report describes the proposed design and implementation plan. We have 
attempted to also describe the main design and implementation alternatives 
available, justify the choice of particular options, and note any important 
limitations of the proposed design. 

We view the Educational Assessment and Accountability System as adaptive 
to users' information needs and evolving over time. Early experience with 
the system, including comments and criticism of the present plan, will help 
guide its future development. We also acknowledge that legitimate 
differences in philosophy and views of what constitutes a "best" approach to 
educational assessment and accountability exist. Thus, the present plan 
should be viewed as a kind of "starting point" from which a future path to 
educational assessment and accountability will develop. 
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Design 



Creating an educational assessment and accountability system requires difficult 
choices on issues involving complex trade-offs. There does not exist an 
established model system that can be simply adopted (Oakes, 1986, p. 8). The 
design problems encountered are as much ecological (that is, organizational, 
structural, and political) as technical. The State Accountability Study Group's 
report. Creating Responsible and Responsive Accountability Systems (U.S. 
Department of Education, Office of Educational Research and Improvement; 
September 1988) emphasizes that "Each State must design an accountability 
strategy that supports its own educational goals and needs and is consistent with 
its own political traditions" (p. vii). 

In order to acquire a more in-depth understanding of the designs and operations 
of accountability systems in other states, we conducted our own review of the 
sample of state reports used by the OERI State Accountability Study Group. 
(Findings of that review are summarized in Educational Assessment & 
Accountability Project Working Paper #1, Review of Educational Accountability 
Systems in Other States : May 1989 [Draft]). We found that accountability 
systems in other states vary greatly in terms of intended purpose and focus, 
kinds and scope of major activities, locus of responsibility for implementing 
activities, extent of public reporting required, and follow-up uses. For example, 
states' purposes for educational accountability systems ranged from ensuring 
individual student's basic skills acquisition to verifying local school or district 
compliance with state education statutes, policies, and regulations. We also 
learned that the "unresolved dilemmas" cited in the OERI report (for example, 
balancing the tension between oversight and improvement, balancing statewide 
comparability with local ownership, and making fair comparisons) are problems 
encountered by other states' educational assessment and accountability programs. 

A. Performance Assessment 

Educational assessment systems are the foundation for educational 
accountability systems. All such assessment systems consist of three 
essential components: the routine collection of information, evaluation of 
that information, and institutional follow-up. Richards (1988) has studied 
educational assessment systems in many states and school districts 
throughout the nation. He describes "three prototypical models that differ 
according to their purpose: monitoring for regulatory compliance, 
monitoring for instructional diagnosis/remediation, and monitoring for 
school performance outputs" (p. 107). Of these three types, monitoring for 
school performance outputs appears most appropriate as the foundation for 
Hawaii's educational accountability system. In the education accountability 
literature, this type of assessment or monitoring system is commonly 
referred to as performance assessment. 
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The functional objective of the performance assessment approach is to 
promote school performance (e.g., student achievement) by focusing 
oversight on school outcomes. Richards (p. 115) notes that: 

"... centralized monitoring systems may be at odds with the 
growing efforts of educational policymakers at the national 
level to replace regulation with accountability. This conflict 
expresses itself in several reform issue areas; for example, 
school based management versus central control.... 
However, performance monitoring, which focuses on 
outputs, is not necessarily antagonistic to these goals. It 
may encourage districts to seek innovations in resource 
allocation and pedagogy in pursuit of higher performance." 

Although the above statement lends support to the selection of 
performance assessment as the foundation for Hawaii's educational 
assessment and accountability system, it should be acknowledged that each 
type of monitoring system has particular limitations and weaknesses. 
Richards (p. 114) cites three potential problems with performance 
monitoring systems: (1) poorly performing schools may simply lack the 
capacity to respond adequately to negative results, leading to decreased 
motivation and, subsequently, to reduced effectiveness of self-improvement 
efforts; (2) "curricular narrowing" can be an unintended effect for schools 
that perceive great pressure to show improved results; and (3) 
comprehensive performance assessment systems are heavily reliant on data 
processing capacity (i.e., computer systems, availability of relevant 
databases, technical expertise and trained personnel). 

The dependence of the performance assessment approach on relatively 
sophisticated data processing capacity points to an important concern about 
feasibility. The beginnings of an educational assessment and accountability 
system for Hawaii's public schools were sketched in Educationa l Assessment 
and Accountability Plan. Technical Report (Savard & Estes, 1988). 
Although largely conceptual, that plan did examine the Department's 
existing data processing capacity in considerable detail. The report also 
suggested ways of integrating data processing for an educational assessment 
and accountability system with existing operations and with currently 
planned expansions in the Department's data processing systems. 

In addition, support by the 1989 Hawaii State Legislature of the 
Department's 1989-91 biennial budget request for developing an educational 
assessment and accountability system is expected to provide adequately for 
system development and initial implementation. Included in the 
appropriation were sufficient resources to meet initial microcomputer-based 
data processing requirements. 
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Indicators: Context-Inp ut-Output Framework 



The basic building blocks of a performance-based educational assessment 
and accountability system are indicators. Indicators are statistics that serve 
as gauges to inform policymakers, educators, and the public about the 
condition of the educational system and how it is changing. The OERI 
State Accountability Study Group's report (1988) notes, however, that 
"Accountability is a blunt tool unless policymakers, educators, and the 
public have information that allows them to determine the likely sources of 
a problem and find clues about how to fix it" (p. 7). Thus, what is needed 
is not just information about distinct elements of the educational system 
but also information about how the relevant elements work together to 
affect educational outcomes. In essence, model building is needed to 
identify the central features of the educational system. Our point of 
departure is a simple, general model of the schooling process as 
exemplified in Figure 1. 

Furthermore, the basic analytical model for educational assessment and 
accountability we have adopted encompasses three general components: 
CONTEXT, INPUT, and OUTPUT. Figure 2 provides a graphic 
presentation of how these components are related. OUTPUT indicators 
such as graduation rates, grade point average, or performance on 
achievement tests gauge the health of public education. These outcome 
measures are the crux of the assessment and accountability system. 
Outcome measures, however, are necessarily viewed in CONTEXT with the 
community setting, family demographics, and the students' readiness for 
school. In addition to context variables, INPUT indicators reflective of 
policy guidelines and the schooling process itself pre examined. Some 
examples of input indicators include course offerings, course requirements, 
staff allocation, teaching methods, and finances. 

It is critical that the educational assessment and accountability system 
include multiple output or outcome measures that range from achievement 
test scores and attendance rates to disciplinary incidents and graduates' 
performance in the first year of college. Multiple output indicators are 
needed in order to represent adequately the overall results of the 
educational system. In addition, a focus on multiple outputs may also help 
to minimize unintended effects such as "curricular narrowing." 

For policymakers, an understanding of relationships between inputs and a 
school's outcomes, as conditioned by the school-community context, will be 
most useful. The intent of using the context-input-output indicators 
framework is for its potential explanatory and diagnostic value, that is, as a 
guide to how the various elements of the educational system might work 
together to affect outcomes. Also, until such time as more complex models 
have been validated, and practical analytic methods for working with them 
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A Basic Analytical Model for Educational Assessment and Accountability 
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have been developed, there is little to be gained by adopting a model more 
complex than the context-input-output framework. 

Numerous criteria for selecting indicators have been proposed (e.g., Cohen, 
1988; OERI, 1988; Blank, 1989). Essentially, the various criteria focus on 
(1) importance and utility, (2) technical quality, and (3) feasibility issues. 

Important and useful educational indicators should be policy-relevant, 
reflecting conditions and trends that are of concern to policymakers and 
that can be acted upon. For utility to policymakers and other audiences, 
indicators should be aligned with long-range state goals, expected effects of 
reform initiatives, knowledge about effective schools, and public perceptions 
of the central features and outcomes of schooling. Only data of high 
technical quality ~ that is, reliable and valid data - should be used for 
indicators. Unreliable or invalid data may well be worse than no data at 
all. A feasible set of indicators should minimize the burdens and costs of 
data collection and analysis. Respondent burden, in particular, can be 
greatly reduced by coordinating data retrieval activities and by relying upon 
already existing data whenever possible. In practice, some trade-offs among 
the indicator selection criteria are necessary. 

Data Sources 

Several existing data sources, together with new sources currently under 
development, should provide a basis of sufficient scope and depth for an 
initial set of indicators. The previous conceptual plan contained an 
annotated list of 37 major data sources and related assessment activities in 
the State Department of Education (see Appendix B of Savard & Estes, 
1988). Of these, the School Profiles is the most comprehensive and the 
single best data source for indicators development. Generated by the 
Department's Information System Services Branch, the School Profiles 
contain multi-year statistical information, updated annually, for each school. 
Drawing upon data extracted from other major data sources, School Profile 
data are available in the areas of student achievement (e.g., Stanford 
Achievement Test Total Reading and Total Mathematics score distributions, 
percent of students with GPAs less than 2.0 or greater than 3.0), student 
demographics (e.g., enrollment, ethnic distribution, percent receiving 
free/reduced lunch, percent limited English proficient, percent certified 
special education), student behavior (e.g., average daily absence, suspensions 
and disciplinary incidents), teacher demographics (e.g., years of experience, 
ethnic distribution), and school-community characteristics (e.g., 
unemployment rate, median income, percent of children from single parent 
homes, percent high school or college graduates). 

Other existing data sources or assessment-related activities that will be 
immediately useful for outcome indicators include: 



• Hawaii State Test of Essential Competencies (HSTEC) 

The HSTEC, a requirement for receiving a high school diploma, is 
administered annually to students in grades 9 to 12. The test 
measures 15 competency areas in the domains of basic and life skills. 
Each essential competency must be passed, but starting in grade 9 a 
student need pass each competency only once. Each year the test is 
administered to about 20,000 students. 

• College Board's Scholastic Aptitude Test (SAT) 

Most of Hawaii's high «chool seniors applying for entrance to 
post-secondary educati a take the College Board's SAT. The 
examination is administered and reported by the College Board. 
About 50% of Hawaii's public school seniors take the test each year. 

• Graduate Follow-Up Survey 

The Graduate Follow-Up Survey examines the educational and 
employment status of Hawaii's public school graduates almost one 
year after high school. Similarity in annual findings have resulted in 
plans to administer the survey every other year beginning with the 
Class of 1989. 

Focus on Schools 

The levels in the educational system at which data can be reported are 
related to the levels at which the data are collected. School-level data can 
be aggregated to form district or state summaries, but it is not often 
possible or feasible to disaggregate data. Although the current design 
focuses on uses of educational assessment and accountability information by 
state policymakers, it seems most desirable to consider individual schools 
the primary units of data collection, analysis, and reporting. 

There is ample research and experience to indicate that individual schools 
are the basic unit of improvement in public education. Thus, the best 
place to observe the interaction of state, district and school policies, as they 
affect students and practitioners, is at the school level. Clearly, the choice 
of a focal level has both technical and policy implications. The unit of 
analysis should be comparable to the level at which policy impact is 
expected; most often, too, this will be the school level. 

Educational assessment and accountability reports, as presently 
conceptualized, would be produced for each school. A state summary of 
information selected from the school reports, together with additional 
information relevant to a state level report (e.g.. National Assessment of 
Educational Progress results, selected results from a new Survey of Public 
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Education in Hawaii), would also be produced. Reporting will emphasize 
the school as the primary unit of accountability, while providing 
policymakers with overall state results. 

The current "School Status and Improvement Report" (previously entitled 
the "School Performance Report Card") is scheduled to be produced 
annually and to be assigned to the Educational Assessment and 
Accountability Project staff. There may be different views of how these 
related efforts can be best integrated. One pragmatic alternative is to 
simply adopt the current "School Status and Improvement Report" as the 
initial school-level assessment and accountability report and, over a period 
of time, adapt that report for congruence with the design concepts outlined 
herein. 

E. Key Questions and Policy Relevance 

Three key questions underlie many recurrent policymaking issues in 
Hawaii's public school system. 

• What are the overall condition, performance, and progress of the 
public schools? 

• What are the accomplishments and shortcomings of the educational 
system? Where are adjustments needed? 

• To what extent are intended effects, including those of major 
education initiatives/reforms, being achieved? 

Fundamental to the present design is the assumption that the resulting 
information will be used with the intent of improving the quality of public 
education in Hawaii. We believe the three key questions cited above serve 
as a central guide to providing policy-relevant information. Answers to 
those questions will necessarily involve several major improvement themes. 
Some examples of these themes include: 

• Increased student achievement, particularly in verbal and higher order 
thinking skills; 

• Improved quality and professionalization of teaching and 
administration; 

• Improved instructional practices; 

• Improved governance and organizational structure for the public 
school system; 
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• Increased role of parents and the community in the public schools; 

• Increased public knowledge and understanding of public education. 

Analysis 

Given a performance assessment approach, indicators data organized in a 
context-input-output framework, individual schools as the primiry units of 
data collection and reporting, and a central focus on state education 
policymaking issues, ho% will these various components be integrated, via 
analysis, into a cohesive assessment/accountability system? The ^alysis 
component of the present design must include three analytic 'unctions 
necessary to operationalize the use of indicators for the purpose of 
providing policy-relevant information about schools' performance: (1) 
describe performance to answer the question "What is happening?"; (2) 
relate performance to inputs and context variables to answer the question 
"Why might it be happening?"; and (3) compare performance to answer the 
question "Is it excellent? adequate? in need of improvement?" 

For descriptive and relational analyses, relatively standard statistical 
techniques and well-developed analytic methodologies are available. Basic 
descriptive statistics (e.g., measures of central tendency, dispersion) 
distribution summaries (e.g., frequency tables, graphs) will ordinarily be 
sufficient to describe performance on output indicators. In order to relate 
outcomes to multiple input and context variables, the methods of analysis 
of variance/covariance, multiple linear regression, and other forms of multi- 
variate analysis will often be applicable. 

Analytical methodologies for comparing schools' performance, however, are 
relatively new, not generally well-established, and often fraught with 
complex technical problems. Further, concerns among practitioners about 
unfair comparisons and the misuse of results can create considerable 
controversy and criticism. The OERI State Accountability Study Group's 
report (1988) cautioned that "One of the most politically troublesome and 
technically sophisticated decisions States make is whether and how to 
compare the performance of schools or districts with one another" (p. 24). 

In concept, several different bases for conducting comparative analysis of 
outcomes can be used. The most common bases for comparison consist of 
contrasting a school's performance with: (1) State goals or standards; (2) 
national norms; (3) the school's past performance; (4) all other schools' 
performance; and (5) similar schools' performance. Most important to note 
is that the various types of comparisons do not yield equivalent 
information. That is, each type of comparison can be viewed as answering 
a different question. For example, where discussion concerns improvement 
over time, comparisons with respect to previous performance are 
meaningful. But that same comparison, while perhaps interesting, would be 
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largely irrelevant to informing a discussion about parity with national 
norms. 

Clearly, different comparisons serve different purposes and these are not, 
generally, interchangeable. There are also practical and technical problems. 
In some cases the needed comparison data are simply nut available. For 
example, education goals are often so generally stated that no measurable 
targets or standards are actually established. As another example, there is 
no agreed upon methodology for constructing comparisons of the "similar" 
schools type. In any event, it is probably unnecessary to generate (even 
were it feasible) all possible comparisons for all outcome indicators. 

However, it must be acknowledged that we currently lack a clear guide as 
to which comparisons for which indicators would be most useful to 
policymakers. We can only exercise our best professional judgement and 
modify that judgement as experience and f eedback accumulate. 

Descriptive, relational, and comparative analyses tend to form a sequence 
of increasing analytic complexity, with later types building on the results 
from earlier ones. We envision the initial years' assessment/accountability 
reports as based largely on the results of descriptive and relational analyses, 
but also containing findings on a few selected comparative analyses of 
outcomes. The comparative analyses, initially, would likely use schools' 
past performance and/or national norms as bases for comparison. 

The capacity of the Educational Assessment and Accountability System to 
generate comparative analyses, in particular, is expected to increase rapidly 
within the first two to three years of operation. By then we should have 
relatively complete collections of descriptive findings, a still growing but 
probably useful set of relational findings, and some experience with and 
feedback from initial comparative analyses. 

Accountability Uses 

The design of the present system utilizes a performance assessment 
approach; it serves as a tool for improving public education by attempting 
to better inform policymakers, educators, and the general public. Simply 
said, the design is outcome-oriented and it seeks improvement by focusing 
attention and discussion on educational outcomes. 

A given assessment and accountability system can, at best, serve one major 
charge. The design of the system outlined herein is not based on a 
compliance monitoring model; it cannot adequately monitor compliance 
with state education statutes, policies, or regulations. Neither is the present 
system based on an instructional diagnostic model; it cannot ensure that 
any one particular student is making satisfactory academic progress. 
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Unlike the compliance monitoring or instructional diagnosis types of 
systems, findings from performance assessment systems do not necessarily 
trigger a specific accountability-related sequence of follow-up actions. 
This can be viewed as either a strength or a weakness. 

By contrast, for sample, findings of non-compliance in a compliance 
monitoring system nearly always result in a fairly fixed sequence of 
follow-up activities: terms and a timeframe to achieve compliance are 
negotiated and formalized in a plan to correct deficiencies; oversight is 
increased; and, following an agreed upon period of time, compliance 
monitoring is again conducted. Note, too, that such a compliance 
monitoring system is well-suited to, and seres to reinforce, an 
organizational setting that could be characterized as centralized and 
bureaucratic. 

The performance assessment approach does permit accountability follow-up 
in either ccntraiized-bureaucratic or decentralized-professional 
organizational settings. It is possible, for example, to devise accountability 
requirements that specify what particular follow-up actions must accompany 
given levels of assessment results. It is also possible that, while no specific 
follow-up action is mandated, the general requirement is that some locally 
determined follow-up action will result. Realistically, the situation is not 
"either-or" and the problem is really one of finding the appropriate balance. 

Unless assessment results are used in some instrumental way for school 
improvement, accountability has not been realized. The following are some 
uses we propose should be made of the assessment reports. 

Use assessment findings/reports to: 

• Provide for accountability to the general public; 

• Provide for accountability to policymakers and to assist the 
development of educational policy; 

• Guide the Department's program planning and budget development; 
and, 

• Guide School Improvement Plan development. 

While the above accountability uses can be realized almost immediately, 
other potential uses would require additional development. Results from 
some other states' assessment/accountability systems are used to reward or 
recognize high performing schools, and to assist and/or intervene in low 
performing schools. Whether such accountability uses are desired for 
Hawaii's public schools and, if so, how to operationalize such uses, would 
require extensive discussion and considerable planning. Such uses are 
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beyond the scope of the present plan, but are mentioned here as a kind of 
"place holder" for future consideration. 

Flexible Im pl e men ta tio n 

There is ample evidence to indicate that ineffective implementation is a 
pervasive problem with many initiatives involving social service programs 
and social policies, and an apparent root cause of the failure of many 
social policies and programs to achieve their intended effects. Berman 
(1980) has studied this problem and noted that: 

Two schools of thought and practice have developed 
regarding the design of implementation strategies. One 
view, which could be called programmed implementation 
assumes that implementation problems can be made 
tolerable, if not eliminated, by careful and explicit 
preprogramming of implementation procedures. The other 
view, which could be called adaptive implementation, holds 
that policy execution can be improved by processes that 
enable initial plans to be adapted to unfolding events and 
decisions" (pp. 205-206). 

Berman further noted that a mismatch between implementation approach 
and context of the situation in which the initiative is to be implemented 
can actually worsen the very implementation problems the approach is 
supposed to circumvent. He also discussed several "situational parameters" 
(pp. 213-220) that could serve as a general guide to the selection and 
design of implementation strategies. While a review of this topic is beyond 
the scope of the present plan, we note that a mix of implementation 
strategies may be most appropriate for the Educational Assessment and 
Accountability System. For those implementation areas that are highly 
dependent upon professional judgement rather than routine application of 
simple decisionmaking rules, adaptive rather than pre-programmed 
strategies will work best. 

Also, we recognize that educational policies and programs are not static; 
they are often "moving targets" that change over time. Michael Cohen, the 
associate director for education programs for the National Governors' 
Association, recommended that: 

"State assessment systems should be adaptive, continuously 
evolving over time. The content, structure, and operational 
procedures of state assessment systems should be modified 
in response to such factors as early experience with the 
system, changing state goals for education, growth in the 
knowledge base regarding effective practices, and natural 
shifts in the attention of state-level policymakers" (1°S8, 
p. 585). 
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Implementation Plan 



The present five-year implementation plan for educational assessment and 
accountability covers the time span 1990-1994. In response to the 
rtcommendation contained in the Legislative Auditor's final report on the 
Evaluation of the Administrative Flexibility Legislation Affecting the Department 
of Education and University of Hawaii (1989), this section describes the plans 
developed by and implementation steps taken by the Department, and joint 
activities conducted by the Department and the University of Hawaii that relate 
directly to educational assessment. 

A. Goals and Objectives 

The following four goals define the functional scope of the Educational 
Assessment and Accountability System. 

• Establish a statewide system of educational assessment and 
accountability to systematically examine the health and quality of 
Hawaii public education. 

• Institute public accountability through periodic reports on public 
education to the community-at-large (parents, businesses, taxpayers). 

• Inform educational policymakers and educators about the condition, 
performance and progress of Hawaii public education. 

• Work collaboratively with the University of Hawaii system to 
coordinate educational assessment activities between the Department 
and higher education. 

Of noteworthy interest are the national education goals drafted by 
President Bush and the nation's governors at the recent education summit 
held on September 28, 1989. The summiteers, including Governor John 
Waihee, agreed to set specific national education goals for a number of 
priority areas such as dropouts, school violence and student achievement 
( Honolulu Advertiser . September 29, 1989). The overall consensus reached 
by the governors and the President is to facilitate restructuring of public 
schools through the establishment of "performance goals" and standards. 

The need to specify desired outcomes is clear and the call for "measurable 
goals" will necessitate further delineation through relatively specific 
objectives. Similarly, consistent with the intent of the four broad goals 
outlined ibove,- a number of major objectives have been drafted for the 
Educational Assessment and Accountability Project (Table 1). 
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Table 1 

Coals »f*d Mtatfor Object i ,es of the 
Educational Assessment and Accountability (EAA) System 
for Hawaii Publk Education 



Goals 



Major Objectives 



Establish a statewide 
system of Educational 
Assessment and 
Accountability to 
systematically examine 
the health and quality of 
Hawaii public education. 



1.1 



Institute puhl ic 
accountability through 
periodic reports on public 
education to the 
communrty-ui-)arge. 



Inform educational 
policymakers and 
educators about the 
condition, performance 
and progress of Hawaii 
public education 



Work collaboratively with 
the University of Hawaii 
system to coordinate 
educational assessment 
activities between the 
Department and higher 
education. 



1.4 



1.? 



I .ft 



1.7 



1.9 



1.10 



1.11 



3.1 



A 2 



3 * 



4.1 



4.3 



4,4 



Planning and Evaluation Branch personnel will monitor and participate in efforts to improve 
educational assessment beginning Spring 1989. 

EAA Project staff will utilize selected measures in the SS1R as interim status indicators for 
the EAA System while developmental work is conducted on an initial set of contcit, input and 
output indicators beginning Fall 1989. 

An Advisory Panel whose membership will consisr of policymakers, educators, parents and the 
commumty-at-large will be established in August 1990 to serve in an advisory capacity in 
overseeing the development of the EAA System. 

Policy makers, educators, and Advisory Panel members will help identity and prioritize policy 
issues and assessment questions (on going). 

EAA Project staff will research and refine an initial set of context, input and output indicator* 
consistent with the thrve purposes of the EAA System by December 1991. 

EAA Project staff will research and identify, with the assistance of a special \d\is<»r> Panel 
subcommittee on standards and an independent consultant, performance criteria necessary to 
institute educational standards in the areas of student achievement and behavior by December 
1991. 

EAA Project staff, with the assistance of an external contractor, will design a draft data 
collection plan to include data definitions, data collection instructions and procedures, quality 
control procedures, documentation for training school-level personnel, and data collection forms 
by October 1992. 

EAA Project staff will design and develop a database management system to store, compile, 
validate, analyze and retrieve EAA information by December IW2 

EAA Project staff will design and develop the analytical methodology required to provide 
descriptive as well as analytical links among educational indicators b> July 1993, 

EAA Project staff will develop the items and format for an EAA schooMevel report thai 
integrates SSiR data for elementary schools by December 1W3. 

EAA Project staff will develop the items and format for an EAA school-level report that 
integrates SSIR data for secondary schools by December 1993. 

EAA Project staff will assist the Department's Public Relations staff in preparing brief 
summaries of statewide EAA data for publication in local newspapers commencing December 
1994. 

EAA Project staff will design and issue a statewide biennial report that clearly and concisely 
summarizes the highlights of the condition, performance and progress of Hawaii public school* 
beginning December 1994, 

EAA Project staff will issue a biennial State Report on EAA for policymakers (bav<»d on the 
SSIRs submitted by individual schools) commencing Spring 19s*0. 

EAA Project staff will research and develop a composite inde* of leading educational indicators 
to serve as an "early warning system" for policymakers and decisionmakers by December IW2 

EAA Project stuff will prepare and issue a biennial statewide rejwrt on the overall condition, 
performance and progress of Hawaii public schools based on a comprehensive educational 
indicator system commencing December 1994. 

EAA Project staff will coordinate efforts with the University of Hawaii Office of Planning ami 
Policy to re-instate Freshmen performance reports generated on a statewide and school-level 
basis beginning July 1989, 

EAA Project staff will work collaboratively with University of Hawaii program officers and 
institutional analysts to obtain demographic and performance data on community college student* 
through the recently implemented Hawaii Community Colleges Student Tracking System 
beginning July 1990. 

EAA Project staff will work collaboratively with University of Ha wan program n'ncers and 
institutional analysis to design and develop a student tracking system for students attending the 
M*noa campus (on-going). 

The Department's Personnel Office will assist in providing evaluative feedback to the Unitrrtiu 
of Hawaii College of Education on first and second-year teacher reeiuits fur Hawaii public 
schools (on-going). 
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B. Guiding Principles 



The following guiding principles serve to further direct the development 
and implementation of assessment/accountability activities: 

• Information produced by the Educational Assessment and 
Accountability System should be relevant to the development and 
revision of educational policy and to the assessment questions directly 
related to policy issues. 

• The results and report formats should be clear and easily understood 
by policymakers, educators and the community-at-large. 

• An independent advisory panel should -be established to assist in the 
development and re iew of the Educational Assessment and 
Accountability Sys :m. 

• The Educational Assessment and Accountability System should be 
based on an analytical model that serves as a conceptual framework 
for identifying, developing and linking a set of educational indicators. 

• Assessments of educational quality should net be based solely on 
single indices of performance such as standardized tests; rather* 
multiple indicators with analytical links among educational input, 
context, and output elements should be used. 

• The Educational Assessment and Accountability System should be 
adaptable, evolving over time to meet future assessment needs 
necessitated by changes in the public education delivery system itself 
(e.g., School/Community-Based Management). 

• The school should serve as the fundamental unit of analysis, consistent 
with its role as the primary agent of change; efforts to disaggregate 
data at the student level should be attempted if deemed necessary 
and costs are not prohibitive. 

• The Educational Assessment and Accountability System should utilize 
and integrate extant data to the fullest extent possible to reduce 
paperwork and data collection burden on school personnel. 

C. Critical Elements 

There are seven critical elements that serve as cornerstones of the 
Educational Assessment and Accountability System: 

• Policy issues and assessment questions 

• Indicator system 

21 
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• Data collection plan 

• Standards 

• Analysis 

• State and school-level reports 

• Informed decisionmaking 

Each is briefly discussed in the following section. Following these 
discussions are descriptions of the system organization, administration, 
implementation schedule, costs, and evaluation. 

1. Policy Issues and Assessment Questions. 

The Educational Assessment and Accountability System is intended to 
be more than a mere collection of data. Foremost among our guiding 
principles is the requirement that information be relevant to 
policymaking and public accountability. Much work needs to be done 
in this area. None of the other states have been entirely successful in 
this regard. 

Assessment questions subsumed under each policy issue area are 
essential to specifying the type and scope of data required. These 
questions are, by necessity, concrete and reflect more specific concerns 
covered broadly by policy. 

The following are examples of policy issues and assessment questions 
that could be addressed. 



Policy Issue #/: Is the scope and sequence of curriculum 

offerings adequate in preparing graduates for 
subsequent study in higher education? 

Assessment Questions: 

1. Wfiat percent of applicants who graduate from Hawaii 
public schools are accepted at Hawaii's 4-year colleges? 

2. Wliat percent of Hawaii public school graduates receive 
advance placement credits, or course waivers as 
freshmen at UH Manoa? 

3. How many Hawaii public school graduates entering the 
UH system are required to enroll in remedial courses in 
language arts? in math? 

4. Wliich indicators best predict freshmen performance in 
the UH system? 
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Policy Issue #2* How extensive are teacher shortages in Hawaii 

public schools? In which program areas are 
these shortages most critical? 

Assessment Questions; 

I. How many courses, aggregated by program areas 

(language arts, math, science, social studies, fine arts, 
vocational education), are being taug/it by teachers out 
of their field of certification? 

Z How many teachers teach in their respective fields of 
certification at least 75% of the time? 

3. How many teacher positions, by program areas, will be 
vacated in the next five years due to anticipated 
retirement? 

4. What percent of the anticipated teacher vacancies over 
the next five years can be filled by UH College of 
Education graduates? 



Clearly, the number of policy issues and assessment questions could 
easily be overwhelming. Policy issues will need to be examined and 
prioritized to keep assessment activities at a manageable level. One 
of the objectives within the present implementation plan will be to 
compile and organize major policy issue areas and assessment 
questions. Some preliminary work in reviewing current policy issues 
needs to be done in the first year of the implementation timeline. 

Initial work in this area will involve a literature review, interviews 
with local policy analysts, and consensus building on policy issue 
priorities. Next, the Educational Assessment and Accountability 
Project staff will conduct a number of in-depth interviews with 
policymakers to identify their major information needs for policy 
development and monitoring. Completion of these activities will 
enable Educational Assessment and Accountability Project staff to 
develop an initial set of prioritized policy issues and assessment 
questions for field-test by the end of the second year. 
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Indicator System. 

Crucial to the establishment of the Educational Assessment and 
Accountability System is the development of an indicator system. 
Figure 3 provides a graphic look at how indicators relate to the basic 
analytical model. 

The Department has already begun some preliminary work on 
reviewing potential indicators. A lengthy yet far from exhaustive list 
of such indicators has been drafted in a form of a taxonomy 
(Educational Assessment and Accountability Project Working Paper 
#2, A Taxonomy of Indicators for Educational Assessment and 
Accountability : August 1989 [Draft]). 

The taxonomy lists potential indicators by assessment domains. Fof 
example, context indicators are organized under two assessment 
domains: demographics (1.10) and early education 
experience/readiness (1.20). "We have underscored potential to 
emphasize that this taxonomy was designed to embrace much more 
than would be feasible in an actual educational assessment and 
accountability system. 

The taxonomy is specifically intended to serve as a resource document 
to guide further efforts toward implementing a useful, representative, 
and parsimonious set of educational indicators. A sample of 
indicators is proposed here (see Table 2), with the understanding that 
a thorough review and field-test will be required for many, if not 
most, of the selected indicators. 

Year 1 in the implementation plan will involve a review and field-test 
of selected indicators. Each of the selected indicators will be 
examined in terms of utility, reliability, and paperwork burden. 
Relationships among the various context, input and output variables 
will also be analyzed where possible. Simultaneously, developmental 
work on other indicators will be conducted if judged useful, feasible 
and promising for policy decisionmaking, public accountability, or 
improving educational practices. 

It should be noted here, that this research and development work will 
be necessary throughout most of the implementation timeline 
proposed herein. Unfortunately, the quality of the data elements are 
occasionally problematic due to definitional ambiguity or variability in 
data collection practices. In point of fact, the quality of educational 
statistics is in need of improvement in ail states, not just Hawaii. 

Several new data sources are currently under development or being 
planned. Some additionally useful educational indicators, based on 



24 



Figure 3 



A Conceptual Framework for Context-Input-Output Indicators Utilizing 
A Basic Analytical Model for Educational Assessment and Accountability 
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Tabic 2 

Sample Set of Initial Indicators 



All Schools 



CONTEXT 



INPUT 



OUTPUT 



% students from single parent homes (census) 
% students with language other than 

English spoken at home 
% SLEP (Levels 1, 2) 
<7c SPED (all certifications) 
% SPED (Mild handicaps only) 
Median family income, community (census) 
% high school graduates, community (census) 
% college graduates, community (census) 
% unemployment* community (census) 
Ethnic composition, student body 
Student transiency rate 



t School organization: elementary, 
secondary, combination 

• Educational initiative/reform implemented: 

SCBM, Year Round schedule 

• School enrollment 

• Student-teacher ratio 

• Expenditures, per pupil 

• Teacher characteristics: 



- age 

- years experience 

- average absences 
Principal turnover index 

% classroom facilities shoit/over 



Academic performance 

- Stanford Achievement Test 

Math 
Reading 

- Grade point average (GPA) 

- No./% grade retentions 
Average daily attendance rate 
Behavior /discipline 

- Index of at-risk behavior, drug & alcohol 
• Suspension rate for type A offenses 

- Suspension rate for type B offenses 

- Arrest rate for school 
"School Climate' 1 survey 

National School Recognition program 



Elementary only 



Secondary only 



• % students on free/reduced lunch program 

• % students with 6 months or more 

pre-school experience 

• 9c EPSS considerable needs 



• % students with family receiving 

Aid to Families with Dependent Childn-n 
or public assistance 



Instructional time allocation (by areas) 
No./% of grade levels participating in 
annual standardized testing 



Course enrollments (by areas) 



• Continuation rate, subsequent enrollment 
in public secondary school 



Academic Performance 

- Natl Assessment of Educational Progress 

- Completion rate 

- % seniors with 20 or more earned credits 

- No./9fc national merit scholars 

semifinatisu 
finalists 

- Scholastic Aptitude Test 

Student satisfaction (graduate follow up) 
Higher Education 

- University of Hawaii "going rates" (proportion 
of students entering UH) 

- Freshman GPA 

No./% vocational education students employed 
in field of training 
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these new data sources, are expected to become available within the 
next one to two years. Most important among the new data sources 
are: 

• University of Hawaii's Freshman Report of High School 
Graduates 

Prior to the implementation of a new computer system in 1986, 
the first year's performance (i.e., college grades and course 
enrollments) of public school students enrolled at the University 
of Hawaii were summarized and reported to each public high 
school. The Department's Planning and Evaluation Branch staff 
is currently working with the UH Office of Institutional Research 
and Office of Admissions and Records to revitalize this report, 
possibly with some expansion of data elements, beginning with 
the high school graduating class of 1989. 

• National Assessment of Educational Progress (NAEP) 

Hawaii, along with 41 other states, the District of Columbia, and 
several territories will participate in NAEPs Trial State 
Assessment Program. Although the program's authorization 
precludes district or school achievement comparisons, the NAEP 
results may be quite useful for state-level assessment reporting. 
Administration of the state-by-state trial testing, limited to 
mathematics in grade 8, is scheduled for February 1990. 

• School Climate/Effective Schooling Survey 

Plans for the development, pilot testing, and statewide 
implementation of a uniform School Climate/Effective Schooling 
Survey have been drafted. The purpose of the survey is to 
describe the effectiveness of several major educational process 
variables as perceived by teachers, students, and parents. The 
survey could provide data for developing additional input 
indicators for educational processes, an area in which too little 
information is currently available (but also an area which poses 
difficult measurement and data collection problems). Contingent 
upon successful development and pilot testing, this project could 
be implemented statewide beginning Spring 1991. 

• Survey of Public Education in Hawaii 

Envisioned as Hawaii's counterpart to the national "Annual 
Gallup Poll of the Public's Attitudes toward the Public Schools" 
(e.g., Elam & Gallup, 1989), the survey would provide for 
systematic monitoring of public confidence and satisfaction with 
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the state's public school system. Selected results would be used 
as output indicators. By including questions from the nationwide 
Gallup Poll, the State can compare the opinions of Hawaii's 
citizens with those of citizens nationally. The survey would also 
provide an opportunity to assess the public's attitudes toward 
special issues (e.g., support for particular local school reform 
efforts). Contingent upon securing the services of a contractor 
to conduct the survey data collection via telephone interviews, 
the project is scheduled to start in Spring 1990. 

• Instructional Practices Surveys (Testing Supplement) 

The NAEP and some states find that collecting short 
supplemental staff surveys with their achievement assessments 
gives useful information. Such surveys could provide data for 
developing additional input indicators for educational processes, 
for example, about students' opportunity to learn the topics 
tested. The correlation of these inputs/processes with 
achievement test results could greatly improve the diagnostic 
utility of assessment information for practitioners and 
policymakers alike. A small-scale trial test of this approach, in 
conjunction with the Department's Statewide Testing Program, is 
under development for piloting in Spring, 1990. 

As work progresses on the development of the initial set of indicators 
and new data sources, a number of measures will be scrutinized 
specifically for potential use as "leading indicators" (Oakes, 1986). 
The concept of a leading indicator (borrowed from econometrics) is 
an intriguing one for educational assessment and accountability. The 
intent of establishing a small array of leading educational indicators is 
to provide a very concise account of the condition and direction of 
public education, and to provide, if possible, an "early warning system" 
for educational policymakers. 

Data Collection Plan. 

In the present document, the term "data collection" is used in its most 
generic sense. It includes specifications of the data definition, 
instrument(s), target respondents, and standard procedures for data 
acquisitions and recording. 

Data Definition. The initial set of indicators will require operational 
definitions for each data element of interest Even seemingly "basic" 
indicators such as attendance rates, graduation rates and dropout rates 
are often inconsistently defined and reported across states, districts, 
and schools. The "Wall Chart" indicators, first introduced by the U.S. 
Department of Education a few years ago, exemplifies this point. 
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Critics noted that those hastily compiled statistics were seriously 
misleading when used to subsequently rank and compare the states. 
In response to the numerous concerns voiced by the educational 
community (over the inequities and distortions created by the "Wall 
Chart" statistics), the Council of Chief State School Officers (CCSSO) 
began work on educational indicators in conjunction with the National 
Center for Educational Statistics (NCES). The CCSSO's aim is to 
derive comparable, standardized definitions for all states within the 
next several years. The staff for Hawaii's Educational Assessment and 
Accountability System will be closely monitoring CCSSO's/NCES* 
activities in this area, and will attempt to integrate existing indicator 
definitions with nationally recommended definitions where appropriate 
and feasible. 

Procedures . Detailed data collection procedures will be drafted 
during the first few years of the implementation plan. As 
developmental work progresses in other related areas (e.g., prioritizing 
policy issues and assessment questions, and operationalizing data 
clement definitions), data collection instruments and procedures will 
be »vfined. Wherever possible, though, additional paperwork burden 
will be minimized by utilizing extant data from existing educational 
assessment activities. 

Implementation . The first step in data collection will be the 
development and definition of an initial set of context, input and 
output measures. Some indicators will be based on singular statistics 
(e.g., number of students receiving free/reduced lunch). Others will 
be composite indicators. Examples of composite indicators include 
stuJent-teacher ratio, average daily attendance, and the U.S. 
Department of Education's index of "educational service requirements" 
(Oakes, 1986; p. 4). 

The second step will involve the development and refinement of data 
collection forms and instruments to obtain information not available 
from existing sources. Data collected will be collated, tabulated and 
stored in computer database files created for the Educational 
Assessment and Accountability System. 

These activities require a third step in the data collection scheme: 
the establishment of a database storage and retrieval system. Initial 
computer hardware has been procured for the Educational Assessment 
and Accountability System in Summer 1989. An IBM PS/2 model 80 
microcomputer with 1 15 megabyte^ of storage capacity will serve as 
the primary data storage unit. Software required to store, retrieve 
and conduct data analyses will be purchased shortly as specifications 
for storage, retrieval, analyses and reporting become clearer. The 
primary data entry methods include keypunching, computerized 

29 



scanning and, where possible, computer tile transfer from the 
Department's Information System Services Branch. The sheer volume 
of some types of student and school data (e.g., course enrollments and 
grades) will require the supplemental use of the State's larger IBM 
mainframe. 1 

The fourth step, and an especially important aspect within the data 
collection design, is a quality assurance component to provide validity 
checks and data quality controls. Poor data quality is currently a 
problem in a number of areas. A set of quality assurance checks will 
need to be developed to encourage accurate data reporting. Such 
quality checks coupled with additional, direct incentives for schools 
should clear up the data collection problems considerably. 

Examples of quality assurance provisions that may be implemented on 
a trial basis include: (1) cross-check counts for figures logically 
related or dependent across different indicators; (2) random audits by 
the Management and Compliance Branch of the Department of 
Education or an independent, external agency of a small proportion of 
schools (5%); (3) development of data processing algorithms to 
identify anomalies in school performance data (e.g., large percentage 
changes, unexpected pattern alterations); and (4) development of 
incentives for complete and accurate reporting. 

Some ensuing steps in the data collection plan are n<v readily or 
easily described at present. There is a potential for computer file 
transfer directly from schools, but the infrastructure required for direct 
electronic transfer is not yet in place. Of special interest is the 
Studen* Information and Program Management System (SIPMS) 
currently under development. It is quite conceivable that the 
Educational Assessment and Accountability System and SIPMS could 
share communication lines and database access. The Financial 
Management System (FMS), too, utilizes a direct school to 
district/state link via a communications network supported by the 
Information System Services Branch (ISSB) within the Department. 



] At this juncture, a caveat is needed to mention plans for reorganization of the 
Hawaii State Department of Education. Especially pertinent to the current discussion 
is the plan to establish a new Office of Information and Technology Services under 
the Superintendent of Education. It is uncertain how the reorganization will affect 
data collection procedures for the Educational Assessment and Accountability System. 
This., again, points to the importance and usefulness of the criteria of system 
adaptability. 
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On the other hand, several additional and supporting activities can be 
clearly anticipated: producing complete documentation of dava 
collection procedures, updating data collection instructions and 
guidelines, and providing related in-service training. Whatever tbc 
outcome of the Student Information and Program Management 
System, or changes in school management practices, such supporting 
activities will be necessary to a successful data collection effort. 

Also envisioned is the complete integration of the School Status and 
Improvement Report (see Appendix B) into the Educational 
Assessment and Accountability System. Measures contained in the 
School Status and Improvement Report can serve as an interim set of 
educational indicators. Future data collection from the school level is 
not anticipated to be much different from that utilized for the current 
School Status and Improvement Report. The current attempt to 
provide schools with forms containing pre-printed data available from 
the Department's Information System Services Branch is of special 
interest. If that attempt encounters no serious difficulties, the 
Educational Assessment and Accountability System will use similar 
procedures. 

Standards. 

Educational standards are uncommon in education today. It may be 
that establishing standards in education has heretofore been viewed an 
impossible task. It is true that the goals of education are many and 
not necessarily consistent, reflecting differences in visions of what the 
future should be. Moreover, there exist a myriad of concomitant 
educational objectives. And there are no agreed upon ways to go 
about setting educational standards. All of these factors contribute to 
great difficulties in establishing standards, but perhaps the most salient 
barrier is the lack of a widely acceptable model of educational 
assessment. 

Why is the absence of an agreed upon assessment model of 
significance? The answer lies in its capability to infuse meaningful 
comparisons within a conceptual framework. An assessment model, in 
essence, provides the structure or "glue" that anchor operational 
activities. Assessment approaches can otherwise take a variety of 
disassociated forms. One could, for example, assess the relative 
improvement or progress of a student, class, school or state over time 
by comparison against a particular baseline measure. Alternately, 
these same units of analysis (student, class, school, state) could be 
evaluated against absolute criteria and then compared with each other. 
The comparisons could also incorporate an ordinal scale to produce 
rankings (essentially what was done in producing the U.S. Department 
of Education's "Wall Chart"). There are other methods, although still 
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experimental in nature, to compare schools with other "similar" 
schools. It is also possible to establish standards using nationally 
norm-referenced or locally norm-referenced criteria. There seems to 
be come current interest in the possibility of establishing standards by 
using expert/professional judgement. 

Some recent developments in standard setting are worth noting. 
Several months prior to the national education summit, Secretary 
Cavazos posed a challenge to the governors of each state to address a 
number of specific educational concerns. Some of these challenges 
employed a minimum standard (e.g., 90% attendance rate), though 
details are lacking on whether such standards are across-the-board, or 
are intended primarily for elementary or secondary levels. Secretary 
Cavazos' standards seem somewhat arbitrarily set, based on national 
data. Nonetheless, these are standards, and should the governors 
choose to accept the challenge, the end result will in effect produce a 
national and state-by-state indicators set that incorporate national 
norms. It appears the governors have accepted the challenge, at least 
in concept, to establish national educational goals and standards 
( Honolulu Advertiser . September 29, 1989). 

A similar challenge was encountered by the Task Force that reviewed 
revisions made to Hawaii's School Performance Report Card in Spring 
and Summer of 1989. The question of standards for outcome 
measures was brought to the attention of Task Force members on 
several occasions. The Task Force's consensus, ultimately, was to 
defer further discussion to another study group more adequately 
prepared to address standards setting issues directly. In retrospect, it 
was probably the most pragmatic and politically astute decision made. 

The area of standards setting is indeed difficult. But the need to 
establish educational standards such as minimally acceptable 
performance expectations (i.e., minimal standards) based on strong 
rationale rather than impulse seems more critical than ever. School, 
district and state officials may not be so adamantly opposed or fearful 
of standards if results are truly useful, and are analyzed cautiously and 
reported fairly. 

The first year of the implementation plan will include preparations for 
convening a rpecial study group on standards setting. Due to the 
difficult and complex nature of the task, the Department will likely 
conduct a nationwide search to acquire the services of professionals 
with expertise in this area. A number of highly reputable institutions, 
including the RAND Corporation and several major universities and 
educational agencies, has expressed a willingness to h lp Hawaii with 
standards setting. Simultaneously, the Educational Assessment and 
Accountability Project staff will initiate work on reviewing pertinent 
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literature on establishing standards during this first year. The second 
and ensuing years will be devoted to preparation of materials 
necessary for use by the special study group, documentation of study 
group recommendations, and field-testing of educational standards for 
selected indicators prior to formal adoption. A key operating 
principle underlying efforts in this area is that of proceeding 
cautiously. This is an exceedingly sensitive issue for many 
stakeholders, and Hawaii's efforts will be pioneering and viewed as an 
example - whether good or bad - by other states. 

Analysis. 

The critical element of establishing educational standards is seen as a 
necessary but insufficient condition for -facilitating analysis of school 
performance. The two critical elements go hand-in-hand, but as 
difficult as standard setting may be, securing analytical whys and 
wherefores must stride a step farther. 

at 

How can the adequacy of a school's performance on a set of outcome 

indicators be operationall y defined? A school's performance is 
obviously affected by the availability of basic resources such as 
personnel, textbooks, and classrooms. However, it is not only the 
amounts but also the ways in which the inputs are utilized that is 
important. For policymakers and educational administrators, the 
allocation, distribution, and organization of resources for improved 
educational outcomes are central concerns. Assessments of school 
performance that aim to provide policy-relevant information must 
focus, at least in part, on educational inputs and input utilization. 

A school's performance can also be affected, positively or negatively, 
by context factors (e.g., socioeconomic status of the school's 
community). Most context factors are ordinarily beyond the 
immediate control of either the school or the school system. 
Nonetheless, context factors differ considerably from school to school 
and do affect educational outcomes. Any fair assessment of a school's 
performance must somehow take into account or control for variations 
in context between schools. 

Clearly, the key problem is to develop an analytic methodology 
capable of relating inputs to outcomes while taking into account 
school-community context factors. 

Proposed Methodology. Given the multidimensional 
context-input-output framework to be used, and considering the largely 
quantitative data to be contained therein, statistical methods for data 
reduction and analysis are required. One proposed analytic approach 
includes the technique of multiple linear regression (Mill). MLR is 
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a general statistical technique through which the relationship between 
a criterion (dependent) variable and a set of predictor (independent) 
variables can be analyzed. For present purposes, the most important 
uses of the technique are: (1) to identify the best sets of context and 
input (independent) variables that contribute to the prediction of each 
given outcome (dependent) variable and to evaluate the accuracy of 
the resulting "prediction" equation; and (2) to control for the 
confounding of context variables in order to evaluate the specific 
contributions of inputs to outcomes. 

Using the MLR analysis technique, two distinct approaches to 
assessment methodology could be developed. The first approach is 
characterized by the use of MLR to obtain clusters of schools grouped 
in terms of their similarity on context variables. Under this approach, 
MLR is employed primarily to obtain the school clusters. The 
technique need not be used to directly compare a school's actual 
outcome values with the "predicted" or "expected" values generated by 
MLR analysis. Instead, a school's actual outcomes are compared with 
the actual values of similar schools in the same cluster. This 
approach, by limiting comparison to similar schools, circumvents the 
often heard criticism about "comparing apples with oranges." 

The second approach is characterized by the use of MLR to take into 
account and adjust for the joint effects of context variables in order to 
generate "expected" values of the outcome indicators. The assessment 
strategy employed in this approach is to then compare actual outcome 
values with MLR-generated "expected" values. No explicit clustering 
of schools is necessary using this approach. Essentially, a statistical 
"leveling of the playing field" is realized within the MLR procedure so 
that the resultant "expected" values have been adjusted for school to 
school context differences. 

Several states currently utilize MLR approaches as part of their 
assessment/accountability systems. A few states (e.g., California, 
South Carolina) rely extensively on variants of the two prototypical 
approaches outlined above. MLR-based approaches, however, are not 
without limitations and problems. 

The major areas of limitations and problems include the following: 
• Technical Limitations 

Results for small schools may be unreliable and can be 
influenced by the unusual performance of only a few 
students. This problem, however, is endemic to all 
statistical analysis of quantitative data and not unique to 
MLR analysis. 
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The precision of MLR-gene rated estimates and the 
confidence that one can have in conclusions derived from 
MLR analysis is directly dependent upon the number of 
cases (V size) available for analysis. Since schools will be 
the basic unit of analysis, and because it may be necessary 
to develop separate sets of regression equations for 
elementary and secondary schools, there is some uncertainty 
as to whether Hawaii's public school system is large enough 
for reliable application of MLR analysis. 

• Utility Problems 

MLR analysis is not easily understood. There is 
considerable evidence, from both research and practical 
experience, to indicate that what decisionmakers and 
policymakers don't understand, they won't use. 

The results of MLR analysis provide school assessment 
information based on a locally norm-referenced comparison. 
That is, the resulting information is of the general form 
"School X's result compared to those from similar Hawaii 
public schools." This may represent an information utility 
problem, and an inherent limitation of MLR analysis, if 
what policymakers need instead is performance assessment 
information of another type (e.g., based on comparisons 
with state standards, national norms, or past results). 

• General Concerns 

A general concern with any analysis that groups similar 
schools together in order to adjust or control for contextual 
differences is the possibility that "self-fulfilling prophecies" 
of low performance expectations might be inadvertently 
created. 

Considering the experience of other states that have used MLR in 
their assessment/accountability systems, together with the potential 
problems and limitations noted above, it seems advisable not to base 
Hawaii's educational assessment methodology on MLR analysis alone. 
Rather, the proposed methodology includes MLR analysis but as only 
one part of a comprehensive sequence of analysis. Following is a 
brief sketch of the proposed assessment methodology. 
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PART 1 - Current Performance Compared to State Standards 



For each outcome indicator, compare a school's current 
performance with the established state standard or criteria. 
Simple, easily understood information is provided by such 
comparisons. The assessment information directly answers the 
basic question Is the school's performance excellent? adequate? 
in need of improvement?" 

Agreed-upon state standards are required. If the standards 
target minimally acceptable levels of performance, no controls or 
adjustments for context differences between schools would be 
necessary. 

PART 2 - Current Performance Compared to Past Performance 

For each outcome indicator, compare a school's current 
performance with its past performance. This assessment 
information directly addresses the question "Is the school's 
performance improving?" 

Multi-year trend data, defined and measured in the same way 
over time, are required. In addition to the performance analysis, 
relational analyses to identify context or input variables that 
might be consistently related to changes in trends can provide 
important clues for decisionmakers and policymakers about 
possible ways to attain further improvements. 

PART 3 ~ Current Performance Compared to That of Similar Schools 

For each outcome indicator, use MLR analysis ("expected" values 
approach) to compare a school's actual performance with its' 
"expected" performance. This assessment information addresses 
the question "Is the school's performance as good as can be 
expected, given its context and inputs?" 

Complete, accurate data for outcome, context, and input 
indicators are required. Conceptually, this assessment 
information should be of considerable interest and value to both 
educators and policymakers because it will provide insight into 
the effects of inputs on schools' outcomes while the effects of 
context variables are held constant. The confidence that should 
be placed in MLR-based assessment results, however, will 
depend upon the extent to which the previously noted limitations 
of MLR analysis actually come into play. 
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The Role of Analysis in Addressing Kev Questions. The proposed 
analytic approaches under current development are, by design, geared 
to address the three key questions presented in Chapfe' II (p. 14). 
Each of the three-part comparisons is essential to addressing the key 
questions. 

The first key question serves as an example. The question is, in 
itself, a three-part inquiry: 

"What is the overall condition , performance, and progress 
of public education?" 

The application of minimum state standards will be highly relevant to 
appraising the condition of public education. In the absence of 
minimal standards, it would be difficult to answer the related 
question: "What is the quality of public education?" 

Similarly, the application of MLR analysis can be useful in appraising 
the adequacy of performance . Though still experimental in nature, 
the MLR technique does attempt to address directly the question, 
"Are the schools' performance as good as can be expected, given their 
context and inputs?" Again, strong emphasis on propriety 
considerations should accompany any interpretations based on MLR. 

Finally, the assessment of school performance over time is most useful 
in determining the progress being made in public education. "Are the 
majority of schools improving?" "In which program areas have there 
been substantial gains (or losses) in educational outcomes?" 

To avoid oversimplifying relational links among the three analytical 
approaches and further assessment questions, it may suffice to say that 
any assessment question will likely be addressed using multiple 
analytic approaches. 

Several preliminary projects were initiated during 1988-89 that will 
assist with development of the system's analytic component. Two of 
these seem noteworthy. First, we obtained a portion of the School 
Profiles data base and have wonducted a fairly large amount of 
exploratory analysis with it. This activity has provided, and continues 
to provide, much practical "hands on" experience. For example, 
difficulties validating the contents of electronically transferred files, 
and problems with ambiguous codes for missing data values were 
encountered and, eventually, resolved. Where several substantively 
overlapping measures are available (e.g., percent of school's 
enrollment receiving free or reduced price meals, percent of school's 
enrollment from families receiving AFDC and/or General Assistance), 
an approach to reviewing the relative technical strengths and 
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weaknesses of alternative measures has been constructed. In terms of 
relational forms of analysis, a number of different methods have been 
tried and some consistent relationships between several context and 
outcome measures were identified. 

Second, we have begun to develop an analytic approach to the 
problem of making comparisons among "similar" schools. Such 
attempts elsewhere (e.g., California, South Carolina) have limitations, 
both technical and in terms of acceptance by educators, that we hope 
to circumvent This effort is currently at the conceptual stage of 
development (outlined in Educational Assessment & Accountability 
Project Working Paper #3, Proposed Method for Assessing School 
Performance ! August 1989 [Draft]). Although promising, the approach 
needs additional refinement, a thorough conceptual critique, and pilot 
testing with Hawaii's data. 

Next steps. There is much exploratory and developmental work to be 
done in this area. The Context-Input-Output analytic model and the. 
establishment of an indicators system provide the basic ingredients 
necessary to initiate the Educational Assessment and Accountability 
System. 

Further developmental work in three areas is needed before it will be 
possible to fully test the assessment methodology. First, the 
development of state standards is required. Second, the data for 
initial sets of school context, input, and outcome indicators must be 
retrieved and compiled into a well-organized database appropriate for 
subsequent analysis. Development of state standards, in particular, is 
expected to be a difficult, time-consuming, and possibly controversial 
project Limiup? the initial scope of this activity to just a few 
indicators, however, would bring early success in standard setting. 

Third, the potential problems and limitations of MLR analysis, cited 
above, need to be carefully investigated with actual data for at least a 
subset of the context, input, and outcome indicators. There is also a 
need to conduct adjunct studies (using other multivariate methods, 
e.g., canonical correlation analysis, multidimensional clustering, path 
analysis) to investigate and determine, if possible, the structural 
relationships amongst the variables being used to ensure that the 
currently proposed methodology does not generate gross 
oversimplifications. Further, there are a number of other potential 
technical problems that only exploratory analysis with actual data are 
likely to reveal. Froblems involving missing data values and poor 
quality data, for example, can affect all types of quantitative analysis. 
Such problems need to be identified and thoroughly understood, and 
practical means for resolving and correcting them need to be 
developed and implemented. 
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The first year implementation activities involving the development of 
an initial set of educational indicators and work on standards setting 
are crucial prerequisites to the proposed analytical methodology. Also 
important are the exploratory studies needed to ascertain the utility of 
a special subset of leading indicators," and the actual performance of 
particular analytic techniques. 

Reporting. 

Reports will be prodded to summarize information at the school and 
state levels. The initial statewide report is scheduled to be released 
in early 1990. The bulk of that report will be based on information 
from the newly implemented School Status and Improvement Report 
(SSIR) [a revised version of the School Performance Report Card]. 
The SSIR is expected to serve as an interim school report prior to 
modification into a more comprehensive Educational Assessment and 
Accountability report format. 

Development of prototype school and state Educational Assessment 
and Accountability reports will begin in the second year of 
implementation. The report formats are expected to be slightly 
different for elementary and secondary schools. 

The first prototype Educational Assessment and Accountability report 
will be developed for elementary schools by mid-1992 (third year of 
implementation). Developmental work on indicators at the secondary 
school level is expected to take longer than at the elementary level. 
The secondary level prototype report is scheduled for completion by 
mid- 1993. After subsequent field-testing with a sample of schools, the 
statewide release of the first elementary and secondary prototype 
reports is scheduled for January 1994. In contrast, state level 
Educational Assessment and Accountability summary reports will be 
published much earlier, initially using SSIR data that will be available 
beginning Fall 1989. The first comprehensive Educational Assessment 
and Accountability State Report should be available in early 1995, 
after both elementary and secondary school level Educational 
Assessment and Accountability reports have been instituted. 

The contents of future state-level Educational Assessment and 
Accountability Report will include summaries of school performance 
by district and state. These summaries will report results from 
analysis that reflects the influence of contextual factors and resource 
utilization in addition to outcomes. The kinds of information 
presented will range from summaries of community demographics and 
school readiness assessments to achievement levels and outcomes 
beyond high school graduation. Same of the most interesting results 

39 



will involve performance assessments of public school graduates and 
findings from those analyses that examine relationships between input 
and outcome measures while controlling for context variables. 

The first few developmental years notwithstanding, the purpose of the 
state-level report is to appraise the condition, performance and 
progress of Hawaii public education. Efforts will be made to identify 
overall trends and to assess the general impact of educational 
initiatives and reforms. Also, a special effort will be made to provide 
information in clear and easily understood formats. 

A general report to the public also will be issued biennially beginning 
1995. Contents of that report will include highlights of educational 
assessment and accountability findings and a special feature section 
covering topics of current interest (e.g., Hawaii opinion poll on public 
education, graduate follow-up, SCBM, etc.) 

Implementation plans also include partial integration with existing 
educational assessment activities such as the Western Association of 
Schools and Colleges (WASC) accreditation. In the second year of 
implementation, educational indicators will be reviewed by the 
Educational Assessment and Accountability Project staff and the 
Advisory Panel to screen potential indicators useful for accreditation 
purposes. Information submitted by schools for accreditation reports, 
at present, consist almost entirely of input or process measures. 
Several context and output indicators, if developed properly, could 
serve as valuable supplements to existing accreditation data. 

Informed Decisionmaking. 

The basic tenet that underlies the educational assessment and 
accountability concept is that overall improvements in education can 
be greatly facilitated by providing key information in the right place 
at the right time. The ultimate utility of information made possible 
th/ough the Educational Assessment and Accountability System is its 
potential for informed decisionmaking at school, district and state 
levels. Parents and the business community, too, should find the 
reports and other end-products useful. 

The state-level, school-level and public accountability reports comprise 
the major deliverables produced by the Educational Assessment and 
Accountability Project, There are, however, a number of other 
possibilities that may benefit decisionmakers and policymakers. Two 
such possibilities are discussed here. 

First, the information presented in the reports will, by necessity, be 
limited to that considered important to the general public or 
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decisionmakers as a whole. The Educational Assessment and 
Accountability database and its affiliated data sources will have the 
potential to provide supplemental information. Confidentiality 
considerations notwithstanding, follow-up information on particular 
relationships among indicators or on special student groups may be 
needed. Special requests for technical assistance or database retrieval 
of information may be a possibility once the basic Educational 
Assessment and Accountability System is in place and data quality is 
improved. 

Second, a number of current or special issue areas will inevitably 
require further in-depth study than is possible with educational 
assessment and accountability information. It may be possible, indeed 
desirable, to conduct special studies on* current educational topics of 
interest For example, there is considerable interest in obtaining a 
comprehensive and accurate assessment of public school graduates' 
employment patterns. The Department of Education, Department of 
Labor and Industrial Relations, University of Hawaii and the business 
community, in particular, would all benefit if employment data on 
recent graduates were made available in a timely manner. 

Another example worthy of investigation is the migration patterns of 
students in and out of the public school system. Present information 
is limited, yet there is much interest in obtaining a clearer picture of 
student movement among Hawaii public and private schools, and 
student flow into and out of Hawaii 

These additional activities are not feasible with current resources, and 
could require some external contracting or access to data sources 
beyond the Department's current capability. These are, however, 
possibilities for policymakers and decisionmakers that may be of 
considerable value in the future. 

Organization and Administration 

1. Organization. 

No major reorganization will be required within the Department of 
Education to implement educational assessment and accountability 
activities. The Educational Assessment and Accountability Project 
staff will be incorporated within the existing Evaluation Section, under 
the Planning and Evaluation Branch, Office of the Superintendent. 
The Planning and Evaluation Branch coord i; ia?s the bulk of existing 
educational assessment activities including s.a*< wide achievement 
testing, PPB-required reports, the current Sviiool Status and 
Improvement Report, and program evaluation. The proximity to 
extant assessment data and collaboration with existing assessment 
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programs should greatly facilitate the accomplishment of Educational 
Assessment and Accountability Project goals and objectives. Figure 4 
shows the organizational linkages among the Educational Assessment 
and Accountability Project, its Advisory Panel, and existing offices. 

2. Administration. 

The administration of the Educational Assessment and Accountability 
Project will be under the general direction of the Evaluation Section's 
Administrator. The current line of authority within the Office of the 
Superintendent will remain intact. However, the entire Department is 
presently being reviewed as pal of an overall reorganization study. It 
is not certain at this time, exactly how such a reorganization will 
ultimately impact upon the proposed educational assessment and 
accountability activities. Hopefully, the reorganization efforts will 
foster more efficient collaboration and sharing of assessment 
information. 

An Advisory Panel on Educational Assessment and Accountability will 
be established in the first year of implementation. This independent 
panel will consist of policymakers, educators, parents and the 
community-at-large. The primary role of the Panel will be to 
periodically review and provide feedback to guide the implementation 
and long-term direction of the Educational Assessment and 
Accountability Project. Resource individuals such as technical 
specialists will be recruited as ex-officio members as necessary. 

Implementation Timeline 

Figure 5 portrays the implementation schedule for major Educational 
Assessment and Accountability tasks anticipated for years 1990-1994. Also 
included, where appropriate, are retroactive 1989 timeline extensions to 
indicate preliminary implementation steps already accomplished. 

Costs 

The following estimates of projected costs are based on the Department of 
Education's 1989-91 biennium budget request document. The 1990-1994 
figures assume an average increase in cost of 5% compounded annually. 
At present, it is extremely difficult to obtain accurate estimuUs of indirect 
costs associated wit^ the proposed Educational Assessment and 
Accountability System since many of the specific activities involving 
personnel from Information System Services Branch, district, school, 
University of Hawaii, Information and Communication Services 
(Department of Budget and Finance), etc., are not known at this time. 
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Much of the detailed information involving in-kind contributions will, in 
effect, be determined by the priorities established for policy issues and 
assessment questions (Major Objective 1.4, Table 1). For similar projects 
of this nature, though, indirect costs have typically ranged between 3-7% 
(5% average) of direct costs. 

The information on costs shown in Table 3 are organized into two major 
sections: Initial costs (Year 1990, [FY 1989-90]) and Per Annum Recurring 
Costs (Years 1991-1994). 

Evaluation 

An objectives-based evaluation will be conducted at the end of each 
implementation year to assess progress being made toward Educational 
Assessment and Accountability Project goals. The evaluation purview will 
be primarily formative in nature with a focus on improving the quality of 
Educational Assessment and Accountability functions. A summary of the 
year's Educational Assessment and Accountability accomplishments, 
problems, and recommendations will be highlighted in an annual report 
prepared by the Educational Assessment and Accountability Project staff. 
The yearly report could be made available to the Legislature by mid- 
session recess (late February). 
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Table 3 

Educational Assessment and Accountability Project 

Cost Estimates 



1. Initial Costs (Year 1990, [FY 1989-90]) 



Personnel 

Evaluation Specialist II 
Evaluation Specialist II 
Secretary II 
Clerk-Typist II 
Student Helper 

Other Curr ent Expenses 
Educational Supplies 
Office Supplies 
Telephone 
Private Car Mileage 
Transportation, Intra-State 
Subsistence, Intra-State 
Hire of Passenger Car 

Rental of Equipment (Copier, Security Alarm) 

Transportation, Out-of-State 

Subsistence, Out-of-State 

Subscriptions 

Registration Fees 

Equipment 

Standard Office Equipment 

IBM PS-2 Model 80 microcomputer, 

software, related peripherals 
Office Furniture for PS-2 system 



Total Direct Costs 

Total Indirect Costs (@ 5% of Direct Costs) 
TOTAL INITIAL COSTS 



34,395 
34^95 
15,532 
13,991 
6.282 
104,595 

250 
2,050 

489 
1,526 
2,000 
1,298 

500 
1,700 
2,000 
1,199 

200 

m 

13,512 

7,548 
10,000 

SQQ 

18^48 

136,455 
6,823 

143,278 
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2. Per Annum Recurring Costs (Years 1991-1994)* 



Personnel 

Evaluation Specialist II 39,215 

Evaluation Specialist II 39,215 

Secretary II 17,707 

Clerk-TVpist II 15,952 

Student Helper fr.2$2 

118,371 

Other Current Expenses 

Educational Supplies 250 

Office Supplies 2,050 

Telephone 489 

Private Car Mileage 1*526 

Repair & Maintenance - Office Equipment 1,027 

Transportation, Intra-State 2,000 

Subsistence, Intra-State 1,298 

Hire of Passenger Car 500 

Rental of Equipment (Copier, Security Alarm) 600 

Transportation, Out-of-State 2,000 

Subsistence, Out-of-State , 1,199 

Subscriptions 200 

Registration Fees 2QQ 

13,439 

Equipment 

Software and software license renewal 600 

Total Average Per Annum Direct Costs 146,677 

Total Average Per Annum Indirect Costs 7,334 
(@ 5% of Direct Costs) 

AVERAGE PER ANNUM RECURRING COSTS 154,011 

TOTAL CUMULATIVE COSTS (Years 1-5) 759,321 



'Figures shown are for FY 1990-91 (Year 1). Assuming an average increase in 
costs of 5% compounded annually, the figures shown above should be multiplied by a 
factor of 1.2155 to obtain the projections for 1994. 
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APPENDIX A 



Examples of Current Educational Assessment Related Activities 
Utilized or Monitored by the Hawaii State Department of Education 



Existing educational assessment related activities: 

1. School profile report 

2. School improvement plan 

3. School climate surveys 

4. School Status and Improvement Report (a.k.a. School performance report 
card) 

5. Accreditation 

6. Authorized Courses and Code Numbers (ACCN) 

7. Longitudinal Student Achievement Monitoring (LSAM) 

8. Computer- Assisted Instruction/Computer Managed Instruction (CAI/CMI) 

9. Hawaii Poll (Honolulu Advertiser, KHON TV) 

10. Graduate follow-up survey, General (40-50% return) 
Graduate follow-up survey, Vocational (10% return) 

11. Freshman Follow-up at University of Hawaii 

12. University of Hawaii Tracking System 

13. Statewide Testing 

• Achievement (Stanford Achievement Test) 

- Reading 

- Math 

- Writing 

• Competency Based Measures 

• Hawaii State Test of Essential Competencies 

• Early Provisions for School Success Program: Missouri KIDS, 
Peabody Picture Vocabulary Test-Revised (PPVT-R) 

• Supplemental Testing: Stanford Achievement Test, Metropolitan 
Achievement Test 
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14. College Board Stanford Achievement Test, American College Testing, 
Preliminary Scholastic Aptitude Test (National Merit Scholars) 

15. College Board Advance Placement exams (college credit) 

16. Vocational Education 

• Pre-Industrial Preparation program (PIP) - Comprehensive Test of 
Basic Skills (approximately 1500) 

17. Students with Limited English Proficiency - English Proficiency Test, 
Metropolitan Achievement Test, Stanford Achievement Test, California 
Achievement Test (n = 10,000) 

18. Special Education - Brigance, Woodcock-Johnson, Metropolitan 
Achievement Test (n = 12,000) 

19. Chapter 1 - Metropolitan Achievement Test, Stanford Achievement Test, 
California Achievement Test (8,000 of estimated 14,000 have pre-post 
data) 

20. Kamehameha programs 

• Center for Development of Early Education (CDEE) 

• Kamehameha School studies 

21. Recognition programs 

• Frito Lay 

• National 

22. Routine Evaluation - 6-year cycle for direct services program 

23. Time allocation study - General Education (manually done) 

24. Crtdit-hy-Exam 

25. Student Achievement Report: Low Achievers, Report to Governor: lowest 
3 stanines and improvement plans 

26. Program Evaluation 
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APPENDIX B 




STATE OF HAWAII 

DEPARTMENT OF EDUCATION 
OFFICE OF THE SUPERINTENDENT 
P.O. Box 2360 
Honolulu. HI 96804 



SCHOOL STATUS AND 
IMPROVEMENT REPORT 



SCHOOL 1 00-AINA HftlNA 



LEVEL li.e K-6, 9- 12) 



K-06 



District HONOLULU 

NUMBER Of STUDENTS 596 



EVALUATOR5 



SCHOOL YEAR: 1988-89 



I. CONTEXT INDICATORS 



A. Students 

Year 

Free/ reduced lunch 
Special Education 
SLEP 

Students enrolled Sept through June 



B. Staff 

# of Principals in last 5 years 

Year 



1987 




1988 




1989 




/ 


% 


/ 


X 


/ 


X 


/ 


X 


/ 


X 


/ 


X 


/ 


X 


/ 


X 


/ 


X 


/ 


X 


/ 


X 


/ 


X 


1987 




1988 




1989 





Total Instructional Staff 
Regular Instruction 
Special Education 
Supplementary Instruction 

Staff with 5* years service in school 

C. Facilities 

1. Number of classrooms available 

Library (% of Ed. Sp#cs> Cafetonum (% of Ed. Specs) Admin Building {% of Ed Specs). 

2 Repairs & Maintenance (work orders) 

# Requests # Fulfilled 

3 Current Facilities Assessment Development Schedule (FADS) available? 
Yes No In Progress 

D. Brief Description of the School 



/ 




/ 


X 


/ 


X 


/ 


X 


/ 


X 


/ 


X 


/ 


X 


/ 


X 


/ 


X 


/ 


X 


/ 


X 


/ 


X 



Number of classrooms short/over 



It. SCHOOL IMPROVEMENT PROCESS 

A. Major Improvement Efforts {as fisted In School Improvement Plans): 



B. Special Programs/Innovative Projects: 



C. Commendations by District Superintendent 



D. Recommendations by District Superintendent:* 
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III. OUTCOMES 






A. Student Achievement: 






Reauired 

Stanford Achievement Test (SAT) Hiahest Grade Tested 






1987 1988 
Total Math / * / * 
Bfilow Avbt aa© (Stanines 1-3) / % / X 
Average (Stanines 4-6) / X / X 
Above Average (Stanines 7-9) 


1989 

/ 
/ 

r 

f 


X 
% 

X 


Total Readina / X / X 
Below A veraae (Stanines 1-3) / X / X 
Average (Stanines 4-6) / X / X 
Above Average (Stanines 7-9) 


1 
f 

/ 


X 
X 
X 


m 

Hiwtn SUte Test of Essential Competencies (HSTECJ / X / X 
Wf% of Seniors Passing) 


/ 


X 


Optional (ea Presidential Fitness, Academic Fitness, #/% of Student Body in the 
4th Quarter with 2.0+ GPA) 

1987 1988 1989 
/ * / * / 
/ * / * 
/ X / X / 


X 

X 
X 


B. Behavioral Data (Chapter 19 Student Misconduct end Discipline) 
A Offenses 






Year 1987 19B8 


1989 




Number of suspensions 

Number of students / X / X 






B Offenses 






Year 1987 1988 


1989 




Number of suspensions 

Number of students / X / X 


/ * 




C. Average Deity Attendence: Year 1987 >988 

X X 


1989 

* 




D. Completion rate: Seniors on count date compared with Seniors who earn diplomas/certificates 

that year. ,987 1988 1989 
Seniors on count date 

Seniors earning diplomas/certificates: / X / X / X 


E. External Reviews (if any; e.g. WASC, NWREL Chap. One and other monitored programs). 
Awards and Recognitions. 


* 
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